2023-12-03T11:12:17.123 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-03T11:12:17.123 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128 --name alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi --description rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} --owner scheduled_alimasa@teuthology --verbose -- /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/orig.config.yaml 2023-12-03T11:12:17.167 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T11:12:17.248 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128 branch: wip-alimasa-multi-pubsub-etag description: rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} email: null first_in_suite: false job_id: '7476128' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-alimasa-multi-pubsub-etag ceph: conf: client: debug rgw: 20 rgw bucket counters cache: true rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= rgw torrent flag: true rgw user counters cache: true setgroup: ceph setuser: ceph global: mon client directed command retry: 5 ms inject socket failures: 5000 osd_max_pg_log_entries: 10 osd_min_pg_log_entries: 10 rgw override bucket index max shards: 1 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 lockdep: true osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 lockdep: true mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) - \(POOL_APP_NOT_ENABLED\) - not have an application enabled - \(OSD_SLOW_PING_TIME sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 rgw: compression type: random ec-data-pool: true erasure_code_profile: crush-failure-domain: osd k: 3 m: 1 name: testprofile frontend: beast inline data: false storage classes: LUKEWARM, FROZEN s3tests: force-branch: ceph-master slow_backend: true storage classes: LUKEWARM, FROZEN thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-alimasa-multi-pubsub-etag sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 owner: scheduled_alimasa@teuthology priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 sleep_before_teardown: 0 suite: rgw suite_branch: wip-alimasa-multi-pubsub-etag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 targets: smithi012.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOYDMCHNBhjGlWzbayJybiaYC93qeZv13jHyTh7c0lvz5KCXrt5k7tAECc6ZHM9rI8bMxWCMauhZfsEvkVYHPEU= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZ+1QdwWJLvtMSWw+KNzg1rIK0cQ9aN/t7GZd/tSC1IjlZ68HUn7gFiinnZ4aJuc/FBuzhStjVN2OcEXlQtDeQ= tasks: - install: extra_system_packages: deb: - s3cmd rpm: - s3cmd - ceph: null - openssl_keys: null - rgw: client.0: null - tox: - client.0 - workunit: clients: client.0: - rgw/run-bucket-check.sh - workunit: clients: client.0: - cls/test_cls_lock.sh - cls/test_cls_log.sh - cls/test_cls_refcount.sh - cls/test_cls_rgw.sh - cls/test_cls_rgw_gc.sh - cls/test_cls_rgw_stats.sh - cls/test_cls_cmpomap.sh - cls/test_cls_2pc_queue.sh - rgw/test_rgw_gc_log.sh - rgw/test_rgw_obj.sh - rgw/test_librgw_file.sh - workunit: clients: client.0: - rgw/test_rgw_s3_mp_reupload.sh - ragweed: client.0: default-branch: ceph-master rgw_server: client.0 stages: prepare,check - workunit: clients: client.0: - rgw/run-reshard.sh - s3tests-java: client.0: force-branch: ceph-master force-repo: https://github.com/ceph/java_s3tests.git - s3tests: client.0: rgw_server: client.0 - workunit: clients: client.0: - rgw/run-versioning.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-03_11:04:43 tube: smithi user: alimasa verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-03T11:12:17.249 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa; will attempt to use it 2023-12-03T11:12:17.249 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks 2023-12-03T11:12:17.249 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-03T11:12:17.251 INFO:teuthology.task.internal:Checking packages... 2023-12-03T11:12:17.277 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bcbf278cf395873c2a482f5a7b2345faf2160319' 2023-12-03T11:12:17.277 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-03T11:12:17.278 INFO:teuthology.packaging:ref: None 2023-12-03T11:12:17.278 INFO:teuthology.packaging:tag: None 2023-12-03T11:12:17.278 INFO:teuthology.packaging:branch: wip-alimasa-multi-pubsub-etag 2023-12-03T11:12:17.278 INFO:teuthology.packaging:sha1: bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:12:17.278 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-alimasa-multi-pubsub-etag 2023-12-03T11:12:17.563 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7537.gbcbf278c 2023-12-03T11:12:17.564 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-03T11:12:17.570 INFO:teuthology.task.internal:no buildpackages task found 2023-12-03T11:12:17.570 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-03T11:12:17.579 INFO:teuthology.task.internal:Saving configuration 2023-12-03T11:12:17.595 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-03T11:12:17.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T11:12:17.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:05:56.491571', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOYDMCHNBhjGlWzbayJybiaYC93qeZv13jHyTh7c0lvz5KCXrt5k7tAECc6ZHM9rI8bMxWCMauhZfsEvkVYHPEU='} 2023-12-03T11:12:17.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:05:56.490166', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZ+1QdwWJLvtMSWw+KNzg1rIK0cQ9aN/t7GZd/tSC1IjlZ68HUn7gFiinnZ4aJuc/FBuzhStjVN2OcEXlQtDeQ='} 2023-12-03T11:12:17.652 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-03T11:12:17.661 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-03T11:12:17.661 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-03T11:12:17.661 INFO:teuthology.run_tasks:Running task console_log... 2023-12-03T11:12:17.732 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f39f43a2f70>, signals=[15]) 2023-12-03T11:12:17.733 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-03T11:12:17.739 INFO:teuthology.task.internal:Opening connections... 2023-12-03T11:12:17.739 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:12:17.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:12:17.838 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:12:17.839 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:12:17.922 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-03T11:12:17.929 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2023-12-03T11:12:17.947 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2023-12-03T11:12:17.947 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="CentOS Stream" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:ID="centos" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE="rhel fedora" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:PLATFORM_ID="platform:el9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:ANSI_COLOR="0;31" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:LOGO="fedora-logo-icon" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://centos.org/" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T11:12:18.005 INFO:teuthology.orchestra.run.smithi012.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T11:12:18.006 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2023-12-03T11:12:18.039 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-12-03T11:12:18.056 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-12-03T11:12:18.057 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-12-03T11:12:18.113 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-03T11:12:18.114 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-03T11:12:18.115 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-12-03T11:12:18.139 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-03T11:12:18.149 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-03T11:12:18.238 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-03T11:12:18.238 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2023-12-03T11:12:18.240 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-12-03T11:12:18.256 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-03T11:12:18.264 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-03T11:12:18.264 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2023-12-03T11:12:18.298 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-12-03T11:12:18.592 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-03T11:12:18.645 INFO:teuthology.run_tasks:Running task kernel... 2023-12-03T11:12:18.661 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-03T11:12:18.661 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-03T11:12:18.662 DEBUG:teuthology.orchestra.run.smithi012:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T11:12:18.662 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-03T11:12:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:12:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:12:18.679 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-12-03T11:12:18.680 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-12-03T11:12:18.737 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-361.el9.x86_64 2023-12-03T11:12:18.737 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-361.el9.x86_64 2023-12-03T11:12:18.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2023-12-03T11:12:18.738 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-361.el9.x86_64 2023-12-03T11:12:18.739 INFO:teuthology.task.kernel:Running kernel on smithi012: 5.14.0-361.el9.x86_64 2023-12-03T11:12:18.739 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum install -y kernel 2023-12-03T11:12:21.267 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 7.4 MB/s | 6.4 MB 00:00 2023-12-03T11:12:21.302 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 7.1 MB/s | 6.4 MB 00:00 2023-12-03T11:12:22.675 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2023-12-03T11:12:22.761 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 5.5 MB/s | 2.0 MB 00:00 2023-12-03T11:12:23.661 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-03T11:12:25.624 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 7.9 MB 00:02 2023-12-03T11:12:27.128 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 18 MB 00:02 2023-12-03T11:12:28.591 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 18 MB 00:01 2023-12-03T11:12:31.911 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 5.8 MB 00:01 2023-12-03T11:12:32.759 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 8.9 MB/s | 5.8 MB 00:00 2023-12-03T11:12:33.387 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 15 kB 00:00 2023-12-03T11:12:34.451 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2023-12-03T11:12:36.422 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-03T11:12:37.825 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 20 MB 00:04 2023-12-03T11:12:41.294 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2023-12-03T11:12:42.770 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-03T11:12:43.650 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T11:12:43.651 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T11:12:43.752 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-03T11:12:43.757 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2023-12-03T11:12:43.758 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2023-12-03T11:12:43.758 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:12:43.758 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 96 M 2023-12-03T11:12:43.758 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 121 M 2023-12-03T11:12:43.758 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-03T11:12:45.202 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-03T11:12:45.202 INFO:teuthology.orchestra.run.smithi012.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-03T11:12:45.284 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout: Package Architecture Version Repository Size 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-03T11:12:45.294 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout:================================================================================ 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout:Install 4 Packages 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:12:45.295 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 96 M 2023-12-03T11:12:45.296 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 121 M 2023-12-03T11:12:45.296 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-03T11:12:45.805 INFO:teuthology.orchestra.run.smithi012.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 20 MB/s | 5.6 MB 00:00 2023-12-03T11:12:45.903 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 3.1 MB/s | 5.6 MB 00:01 2023-12-03T11:12:46.113 INFO:teuthology.orchestra.run.smithi012.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 34 MB/s | 20 MB 00:00 2023-12-03T11:12:46.562 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 8.2 MB/s | 20 MB 00:02 2023-12-03T11:12:46.614 INFO:teuthology.orchestra.run.smithi012.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 35 MB/s | 38 MB 00:01 2023-12-03T11:12:47.006 INFO:teuthology.orchestra.run.smithi012.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 27 MB/s | 32 MB 00:01 2023-12-03T11:12:47.006 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:12:47.006 INFO:teuthology.orchestra.run.smithi012.stdout:Total 56 MB/s | 96 MB 00:01 2023-12-03T11:12:47.370 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 12 MB/s | 38 MB 00:03 2023-12-03T11:12:47.895 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-03T11:12:48.311 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-03T11:12:48.311 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-03T11:12:49.665 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-03T11:12:49.665 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-03T11:12:50.703 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 6.7 MB/s | 32 MB 00:04 2023-12-03T11:12:50.703 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:12:50.704 INFO:teuthology.orchestra.run.smithi191.stdout:Total 14 MB/s | 96 MB 00:06 2023-12-03T11:12:51.561 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-03T11:12:51.934 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-03T11:12:51.934 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-03T11:12:52.469 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-03T11:12:53.138 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:12:53.238 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-03T11:12:53.239 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-03T11:12:53.281 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:12:55.943 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-03T11:12:56.149 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:12:56.335 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:12:56.605 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:12:56.738 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:12:59.489 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:12:59.658 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:13:01.314 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:13:01.435 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:04.229 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:13:04.433 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:06.414 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:36.832 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:36.876 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:37.289 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:37.290 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:13:37.290 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:13:37.290 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T11:13:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:13:37.707 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-03T11:13:38.192 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:13:38.237 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:38.281 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:38.343 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:38.343 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:13:38.343 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T11:13:38.343 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T11:13:38.343 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T11:13:38.343 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-03T11:13:38.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi012.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T11:13:38.344 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:13:38.467 INFO:teuthology.orchestra.run.smithi012.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:38.467 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:13:38.468 DEBUG:teuthology.orchestra.run.smithi012:> sudo rpm -qi grub2-tools 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Name : grub2-tools 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Epoch : 1 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Version : 2.06 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Release : 68.el9 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: x86_64 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T11:13:38.515 INFO:teuthology.orchestra.run.smithi012.stdout:Group : Unspecified 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Size : 8264254 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:License : GPLv3+ 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Packager : builder@centos.org 2023-12-03T11:13:38.516 INFO:teuthology.orchestra.run.smithi012.stdout:Vendor : CentOS 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:Summary : Support tools for GRUB. 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:Description : 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:hardware devices. 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:13:38.517 INFO:teuthology.orchestra.run.smithi012.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T11:13:38.518 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T11:13:38.518 INFO:teuthology.task.kernel:Updating grub on smithi012 to boot 5.14.0-388.el9.x86_64 2023-12-03T11:13:38.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T11:13:38.728 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:38.728 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-03T11:13:38.728 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-03T11:13:38.728 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-03T11:13:39.109 INFO:teuthology.orchestra.run.smithi012.stderr:Generating grub configuration file ... 2023-12-03T11:13:39.145 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-03T11:13:39.145 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:13:39.145 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-03T11:13:39.145 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.145 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.146 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.146 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.146 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:13:39.146 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-03T11:13:39.672 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:13:39.858 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.859 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:13:39.859 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-03T11:13:39.859 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-03T11:13:39.859 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-03T11:13:39.859 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-03T11:13:39.859 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2023-12-03T11:13:39.859 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2023-12-03T11:13:39.977 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-03T11:13:39.977 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-03T11:13:39.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2023-12-03T11:13:40.011 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2023-12-03T11:13:40.011 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2023-12-03T11:13:40.011 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 68.el9 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8264254 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2023-12-03T11:13:40.012 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:13:40.013 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2023-12-03T11:13:40.014 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-03T11:13:40.014 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-388.el9.x86_64 2023-12-03T11:13:40.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-03T11:13:40.446 INFO:teuthology.orchestra.run.smithi012.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T11:13:40.474 INFO:teuthology.orchestra.run.smithi012.stderr:done 2023-12-03T11:13:40.476 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-12-03T11:13:40.491 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.Y7Ck7BFLf3 2023-12-03T11:13:40.492 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Y7Ck7BFLf3 2023-12-03T11:13:40.580 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.Y7Ck7BFLf3 2023-12-03T11:13:40.652 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2023-12-03T11:13:40.689 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.Y7Ck7BFLf3 is 6KB 2023-12-03T11:13:40.736 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.Y7Ck7BFLf3 2023-12-03T11:13:40.751 DEBUG:teuthology.orchestra.run.smithi012:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T11:13:40.839 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T11:13:40.839 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T11:13:40.839 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T11:13:40.839 INFO:teuthology.orchestra.run.smithi012.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T11:13:40.840 DEBUG:teuthology.orchestra.run.smithi012:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T11:13:40.945 DEBUG:teuthology.orchestra.run.smithi012:> sudo shutdown -r now 2023-12-03T11:13:41.973 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-03T11:13:42.004 INFO:teuthology.orchestra.run.smithi191.stderr:done 2023-12-03T11:13:42.005 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-12-03T11:13:42.020 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.Gpvcbfbdv5 2023-12-03T11:13:42.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Gpvcbfbdv5 2023-12-03T11:13:42.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.Gpvcbfbdv5 2023-12-03T11:13:42.229 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.Gpvcbfbdv5 is 6KB 2023-12-03T11:13:42.248 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.Gpvcbfbdv5 2023-12-03T11:13:42.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2023-12-03T11:13:42.326 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-03T11:13:42.326 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-03T11:13:42.326 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-03T11:13:42.326 INFO:teuthology.orchestra.run.smithi191.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-03T11:13:42.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-03T11:13:42.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2023-12-03T11:14:10.949 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-03T11:14:10.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:14:10.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:14:12.486 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-03T11:14:12.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:14:12.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:14:29.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-12-03T11:14:30.882 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2023-12-03T11:14:38.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:14:38.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:14:39.884 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:14:39.885 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:14:42.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2023-12-03T11:14:54.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:14:54.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:14:56.738 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.12 2023-12-03T11:15:08.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:15:08.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:15:09.084 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-12-03T11:15:09.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:15:09.582 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-03T11:15:09.582 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2023-12-03T11:15:09.597 INFO:teuthology.orchestra.run.smithi012.stdout:5.14.0-388.el9.x86_64 2023-12-03T11:15:09.597 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T11:15:09.598 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T11:15:09.598 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T11:15:10.599 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-03T11:15:10.599 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T11:15:10.697 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2023-12-03T11:15:10.737 DEBUG:teuthology.parallel:result is None 2023-12-03T11:15:26.817 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-12-03T11:15:27.200 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:15:27.200 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-388.el9.x86_64"... 2023-12-03T11:15:27.200 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-12-03T11:15:27.214 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-388.el9.x86_64 2023-12-03T11:15:27.214 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-03T11:15:27.214 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-03T11:15:27.215 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-03T11:15:28.215 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-03T11:15:28.215 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-03T11:15:28.272 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2023-12-03T11:15:28.310 DEBUG:teuthology.parallel:result is None 2023-12-03T11:15:28.311 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-03T11:15:31.490 INFO:teuthology.task.internal:Creating test directory... 2023-12-03T11:15:31.490 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-03T11:15:31.493 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-03T11:15:31.511 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-03T11:15:31.517 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-03T11:15:31.528 INFO:teuthology.task.internal:Creating archive directory... 2023-12-03T11:15:31.529 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-03T11:15:31.554 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-03T11:15:31.585 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-03T11:15:31.592 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-03T11:15:31.592 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-03T11:15:31.623 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-03T11:15:31.656 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:15:31.669 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:15:31.669 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:15:31.686 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-03T11:15:31.688 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-03T11:15:31.696 INFO:teuthology.task.internal:Configuring sudo... 2023-12-03T11:15:31.696 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-03T11:15:31.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-03T11:15:31.738 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-03T11:15:31.747 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-03T11:15:31.748 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-03T11:15:31.773 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-03T11:15:31.793 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:15:31.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:15:31.928 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:15:32.013 DEBUG:teuthology.orchestra.run.smithi012:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:15:32.060 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:15:32.060 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-03T11:15:32.131 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:15:32.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-03T11:15:32.225 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:15:32.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-03T11:15:32.347 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:15:32.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-03T11:15:32.415 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2023-12-03T11:15:32.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-12-03T11:15:32.475 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T11:15:32.507 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T11:15:32.730 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-03T11:15:32.738 INFO:teuthology.task.internal:Starting timer... 2023-12-03T11:15:32.738 INFO:teuthology.run_tasks:Running task pcp... 2023-12-03T11:15:32.777 INFO:teuthology.run_tasks:Running task selinux... 2023-12-03T11:15:32.785 DEBUG:teuthology.orchestra.run.smithi012:> sudo service auditd rotate 2023-12-03T11:15:32.873 INFO:teuthology.orchestra.run.smithi012.stdout:Rotating logs: 2023-12-03T11:15:32.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-12-03T11:15:32.928 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2023-12-03T11:15:32.930 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-03T11:15:32.930 DEBUG:teuthology.orchestra.run.smithi012:> /usr/sbin/getenforce 2023-12-03T11:15:32.971 INFO:teuthology.orchestra.run.smithi012.stdout:Permissive 2023-12-03T11:15:32.971 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-12-03T11:15:32.996 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-12-03T11:15:32.996 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi012.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2023-12-03T11:15:32.996 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T11:15:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:15:33.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T11:15:33.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:15:33.072 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-03T11:15:33.072 DEBUG:teuthology.orchestra.run.smithi012:> sudo /usr/sbin/setenforce permissive 2023-12-03T11:15:33.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-12-03T11:15:33.154 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-03T11:15:33.165 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-03T11:15:33.169 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-03T11:15:33.305 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-03T11:15:33.321 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-03T11:15:33.323 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-03T11:21:38.287 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2023-12-03T11:21:38.288 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:21:38.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:21:38.386 DEBUG:teuthology.orchestra.run.smithi012:> true 2023-12-03T11:21:38.457 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi012.front.sepia.ceph.com' 2023-12-03T11:21:38.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:21:38.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-03T11:21:38.543 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-12-03T11:21:38.608 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-12-03T11:21:38.608 INFO:teuthology.run_tasks:Running task clock... 2023-12-03T11:21:38.619 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-03T11:21:38.619 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-03T11:21:38.620 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:21:38.622 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-03T11:21:38.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T11:21:38.656 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-03T11:21:38.678 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-03T11:21:38.678 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-03T11:21:38.694 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-03T11:21:38.715 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: ntpd: command not found 2023-12-03T11:21:38.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-12-03T11:21:38.731 INFO:teuthology.orchestra.run.smithi012.stdout:506 Cannot talk to daemon 2023-12-03T11:21:38.737 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-12-03T11:21:38.751 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-03T11:21:38.754 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-03T11:21:38.770 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-03T11:21:38.771 INFO:teuthology.orchestra.run.smithi012.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-03T11:21:38.813 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.816 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.818 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.822 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T11:21:38.823 INFO:teuthology.run_tasks:Running task install... 2023-12-03T11:21:38.831 DEBUG:teuthology.task.install:project ceph 2023-12-03T11:21:38.831 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}} 2023-12-03T11:21:38.831 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'} 2023-12-03T11:21:38.831 INFO:teuthology.task.install:Using flavor: default 2023-12-03T11:21:38.837 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-03T11:21:38.837 INFO:teuthology.task.install:extra packages: [] 2023-12-03T11:21:38.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'tag': None, 'wait_for_package': False} 2023-12-03T11:21:38.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:21:38.839 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'tag': None, 'wait_for_package': False} 2023-12-03T11:21:38.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:21:38.982 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/ 2023-12-03T11:21:38.982 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7537.gbcbf278c 2023-12-03T11:21:38.987 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/ 2023-12-03T11:21:38.987 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7537.gbcbf278c 2023-12-03T11:21:39.092 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-03T11:21:39.092 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:21:39.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-03T11:21:39.119 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-alimasa-multi-pubsub-etag/bcbf278cf395873c2a482f5a7b2345faf2160319/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-03T11:21:39.119 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:21:39.119 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-03T11:21:39.120 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, s3cmd, s3cmd, s3cmd on remote rpm x86_64 2023-12-03T11:21:39.121 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bcbf278cf395873c2a482f5a7b2345faf2160319/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-03T11:21:39.153 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, s3cmd, s3cmd, s3cmd, s3cmd on remote rpm x86_64 2023-12-03T11:21:39.153 DEBUG:teuthology.orchestra.run.smithi012:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bcbf278cf395873c2a482f5a7b2345faf2160319/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-03T11:21:39.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-03T11:21:39.225 DEBUG:teuthology.orchestra.run.smithi012:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-03T11:21:39.268 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-03T11:21:39.313 DEBUG:teuthology.orchestra.run.smithi012:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-03T11:21:39.335 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2023-12-03T11:21:39.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2023-12-03T11:21:39.342 INFO:teuthology.orchestra.run.smithi012.stdout:check_obsoletes = 1 2023-12-03T11:21:39.344 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum clean all 2023-12-03T11:21:39.580 INFO:teuthology.orchestra.run.smithi191.stdout:73 files removed 2023-12-03T11:21:39.597 INFO:teuthology.orchestra.run.smithi012.stdout:73 files removed 2023-12-03T11:21:39.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd s3cmd s3cmd s3cmd s3cmd 2023-12-03T11:21:39.627 DEBUG:teuthology.orchestra.run.smithi012:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd s3cmd s3cmd s3cmd s3cmd 2023-12-03T11:21:40.344 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 187 kB/s | 78 kB 00:00 2023-12-03T11:21:40.479 INFO:teuthology.orchestra.run.smithi012.stdout:ceph packages for x86_64 152 kB/s | 78 kB 00:00 2023-12-03T11:21:40.770 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-12-03T11:21:40.854 INFO:teuthology.orchestra.run.smithi012.stdout:ceph noarch packages 29 kB/s | 9.5 kB 00:00 2023-12-03T11:21:41.112 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2023-12-03T11:21:41.271 INFO:teuthology.orchestra.run.smithi012.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2023-12-03T11:21:41.683 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-03T11:21:41.748 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-03T11:21:43.066 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2023-12-03T11:21:43.141 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2023-12-03T11:21:43.873 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 7.9 MB 00:00 2023-12-03T11:21:45.524 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 7.9 MB 00:02 2023-12-03T11:21:46.732 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-12-03T11:21:48.109 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 12 MB/s | 18 MB 00:01 2023-12-03T11:21:52.508 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 5.8 MB 00:02 2023-12-03T11:21:53.243 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 5.8 MB 00:01 2023-12-03T11:21:54.778 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 15 kB 00:00 2023-12-03T11:21:55.025 INFO:teuthology.orchestra.run.smithi012.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 15 kB 00:01 2023-12-03T11:21:56.316 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2023-12-03T11:21:59.246 INFO:teuthology.orchestra.run.smithi012.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 20 MB 00:04 2023-12-03T11:22:01.089 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-03T11:22:03.794 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:22:03.794 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:22:03.864 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-12-03T11:22:03.882 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-03T11:22:03.882 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-12-03T11:22:03.882 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-03T11:22:03.882 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 6.6 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 5.2 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 860 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 141 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 1.5 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 133 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 1.8 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 7.4 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 50 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 8.6 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 41 M 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 262 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 774 k 2023-12-03T11:22:03.883 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 31 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 718 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 126 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 162 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 320 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 298 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 99 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 85 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 170 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: s3cmd noarch 2.3.0-4.el9 epel 197 k 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.3 M 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-12-03T11:22:03.884 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 19 M 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 28 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 2.1 M 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 243 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.4 M 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 18 M 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 15 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 24 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-03T11:22:03.885 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 164 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 475 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.8 M 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-03T11:22:03.886 INFO:teuthology.orchestra.run.smithi191.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 45 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 129 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-03T11:22:03.887 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-03T11:22:03.888 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-03T11:22:03.889 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-03T11:22:03.890 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout:============================================================================================ 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout:Install 118 Packages 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2023-12-03T11:22:03.891 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:03.892 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 181 M 2023-12-03T11:22:03.892 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-12-03T11:22:04.200 INFO:teuthology.orchestra.run.smithi012.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2023-12-03T11:22:04.690 INFO:teuthology.orchestra.run.smithi191.stdout:(1/120): ceph-18.0.0-7537.gbcbf278c.el9.x86_64. 40 kB/s | 6.6 kB 00:00 2023-12-03T11:22:04.907 INFO:teuthology.orchestra.run.smithi191.stdout:(2/120): ceph-fuse-18.0.0-7537.gbcbf278c.el9.x8 3.9 MB/s | 860 kB 00:00 2023-12-03T11:22:04.957 INFO:teuthology.orchestra.run.smithi191.stdout:(3/120): ceph-immutable-object-cache-18.0.0-753 2.8 MB/s | 141 kB 00:00 2023-12-03T11:22:05.224 INFO:teuthology.orchestra.run.smithi191.stdout:(4/120): ceph-mds-18.0.0-7537.gbcbf278c.el9.x86 7.8 MB/s | 2.1 MB 00:00 2023-12-03T11:22:05.433 INFO:teuthology.orchestra.run.smithi191.stdout:(5/120): ceph-mgr-18.0.0-7537.gbcbf278c.el9.x86 7.0 MB/s | 1.5 MB 00:00 2023-12-03T11:22:05.759 INFO:teuthology.orchestra.run.smithi191.stdout:(6/120): ceph-base-18.0.0-7537.gbcbf278c.el9.x8 4.2 MB/s | 5.2 MB 00:01 2023-12-03T11:22:05.992 INFO:teuthology.orchestra.run.smithi191.stdout:(7/120): ceph-mon-18.0.0-7537.gbcbf278c.el9.x86 7.9 MB/s | 4.4 MB 00:00 2023-12-03T11:22:06.301 INFO:teuthology.orchestra.run.smithi191.stdout:(8/120): ceph-common-18.0.0-7537.gbcbf278c.el9. 11 MB/s | 19 MB 00:01 2023-12-03T11:22:06.376 INFO:teuthology.orchestra.run.smithi191.stdout:(9/120): ceph-selinux-18.0.0-7537.gbcbf278c.el9 321 kB/s | 24 kB 00:00 2023-12-03T11:22:06.910 INFO:teuthology.orchestra.run.smithi191.stdout:(10/120): ceph-radosgw-18.0.0-7537.gbcbf278c.el 9.4 MB/s | 8.6 MB 00:00 2023-12-03T11:22:06.945 INFO:teuthology.orchestra.run.smithi012.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:22:06.946 INFO:teuthology.orchestra.run.smithi012.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-03T11:22:06.969 INFO:teuthology.orchestra.run.smithi191.stdout:(11/120): libcephfs-devel-18.0.0-7537.gbcbf278c 532 kB/s | 31 kB 00:00 2023-12-03T11:22:07.019 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-03T11:22:07.036 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-03T11:22:07.036 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-03T11:22:07.036 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 6.6 k 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 5.2 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 860 k 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 141 k 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 1.5 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 133 k 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 1.8 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 7.4 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 50 k 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 8.6 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 41 M 2023-12-03T11:22:07.037 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 262 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 774 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 31 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 718 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 126 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 162 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 320 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 298 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 99 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 85 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 170 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: s3cmd noarch 2.3.0-4.el9 epel 197 k 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrading: 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.3 M 2023-12-03T11:22:07.038 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 3.0 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 19 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 28 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 2.1 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 243 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.4 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 18 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7537.gbcbf278c.el9 ceph-noarch 15 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 24 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-03T11:22:07.039 INFO:teuthology.orchestra.run.smithi012.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 164 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 475 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2 x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 4.8 M 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-03T11:22:07.040 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 45 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common x86_64 2:18.0.0-7537.gbcbf278c.el9 ceph 129 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-03T11:22:07.041 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-03T11:22:07.042 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-03T11:22:07.043 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-03T11:22:07.044 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================================ 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout:Install 118 Packages 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout:Upgrade 2 Packages 2023-12-03T11:22:07.045 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:22:07.046 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 181 M 2023-12-03T11:22:07.046 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-03T11:22:07.111 INFO:teuthology.orchestra.run.smithi191.stdout:(12/120): libcephfs2-18.0.0-7537.gbcbf278c.el9. 4.9 MB/s | 718 kB 00:00 2023-12-03T11:22:07.186 INFO:teuthology.orchestra.run.smithi191.stdout:(13/120): libcephsqlite-18.0.0-7537.gbcbf278c.e 2.1 MB/s | 164 kB 00:00 2023-12-03T11:22:07.261 INFO:teuthology.orchestra.run.smithi191.stdout:(14/120): librados-devel-18.0.0-7537.gbcbf278c. 1.7 MB/s | 126 kB 00:00 2023-12-03T11:22:07.354 INFO:teuthology.orchestra.run.smithi191.stdout:(15/120): libradosstriper1-18.0.0-7537.gbcbf278 5.1 MB/s | 475 kB 00:00 2023-12-03T11:22:07.635 INFO:teuthology.orchestra.run.smithi012.stdout:(1/120): ceph-18.0.0-7537.gbcbf278c.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2023-12-03T11:22:07.979 INFO:teuthology.orchestra.run.smithi191.stdout:(16/120): librgw2-18.0.0-7537.gbcbf278c.el9.x86 7.6 MB/s | 4.8 MB 00:00 2023-12-03T11:22:08.063 INFO:teuthology.orchestra.run.smithi191.stdout:(17/120): python3-ceph-argparse-18.0.0-7537.gbc 542 kB/s | 45 kB 00:00 2023-12-03T11:22:08.185 INFO:teuthology.orchestra.run.smithi012.stdout:(2/120): ceph-fuse-18.0.0-7537.gbcbf278c.el9.x8 1.5 MB/s | 860 kB 00:00 2023-12-03T11:22:08.262 INFO:teuthology.orchestra.run.smithi012.stdout:(3/120): ceph-immutable-object-cache-18.0.0-753 1.8 MB/s | 141 kB 00:00 2023-12-03T11:22:08.288 INFO:teuthology.orchestra.run.smithi191.stdout:(18/120): ceph-osd-18.0.0-7537.gbcbf278c.el9.x8 7.0 MB/s | 18 MB 00:02 2023-12-03T11:22:08.305 INFO:teuthology.orchestra.run.smithi191.stdout:(19/120): python3-ceph-common-18.0.0-7537.gbcbf 532 kB/s | 129 kB 00:00 2023-12-03T11:22:08.405 INFO:teuthology.orchestra.run.smithi191.stdout:(20/120): python3-cephfs-18.0.0-7537.gbcbf278c. 1.4 MB/s | 162 kB 00:00 2023-12-03T11:22:08.431 INFO:teuthology.orchestra.run.smithi191.stdout:(21/120): python3-rados-18.0.0-7537.gbcbf278c.e 2.5 MB/s | 320 kB 00:00 2023-12-03T11:22:08.523 INFO:teuthology.orchestra.run.smithi191.stdout:(22/120): python3-rbd-18.0.0-7537.gbcbf278c.el9 2.5 MB/s | 298 kB 00:00 2023-12-03T11:22:08.548 INFO:teuthology.orchestra.run.smithi191.stdout:(23/120): python3-rgw-18.0.0-7537.gbcbf278c.el9 848 kB/s | 99 kB 00:00 2023-12-03T11:22:08.616 INFO:teuthology.orchestra.run.smithi191.stdout:(24/120): rbd-fuse-18.0.0-7537.gbcbf278c.el9.x8 917 kB/s | 85 kB 00:00 2023-12-03T11:22:08.904 INFO:teuthology.orchestra.run.smithi012.stdout:(4/120): ceph-mds-18.0.0-7537.gbcbf278c.el9.x86 3.3 MB/s | 2.1 MB 00:00 2023-12-03T11:22:09.008 INFO:teuthology.orchestra.run.smithi191.stdout:(25/120): rbd-mirror-18.0.0-7537.gbcbf278c.el9. 6.5 MB/s | 3.0 MB 00:00 2023-12-03T11:22:09.083 INFO:teuthology.orchestra.run.smithi191.stdout:(26/120): ceph-grafana-dashboards-18.0.0-7537.g 375 kB/s | 28 kB 00:00 2023-12-03T11:22:09.159 INFO:teuthology.orchestra.run.smithi191.stdout:(27/120): ceph-mgr-cephadm-18.0.0-7537.gbcbf278 1.7 MB/s | 133 kB 00:00 2023-12-03T11:22:09.251 INFO:teuthology.orchestra.run.smithi191.stdout:(28/120): rbd-nbd-18.0.0-7537.gbcbf278c.el9.x86 268 kB/s | 170 kB 00:00 2023-12-03T11:22:09.337 INFO:teuthology.orchestra.run.smithi012.stdout:(5/120): ceph-mgr-18.0.0-7537.gbcbf278c.el9.x86 3.4 MB/s | 1.5 MB 00:00 2023-12-03T11:22:09.501 INFO:teuthology.orchestra.run.smithi191.stdout:(29/120): ceph-mgr-dashboard-18.0.0-7537.gbcbf2 5.1 MB/s | 1.8 MB 00:00 2023-12-03T11:22:09.804 INFO:teuthology.orchestra.run.smithi012.stdout:(6/120): ceph-base-18.0.0-7537.gbcbf278c.el9.x8 2.1 MB/s | 5.2 MB 00:02 2023-12-03T11:22:09.910 INFO:teuthology.orchestra.run.smithi191.stdout:(30/120): ceph-test-18.0.0-7537.gbcbf278c.el9.x 12 MB/s | 41 MB 00:03 2023-12-03T11:22:09.935 INFO:teuthology.orchestra.run.smithi191.stdout:(31/120): ceph-mgr-modules-core-18.0.0-7537.gbc 560 kB/s | 243 kB 00:00 2023-12-03T11:22:09.969 INFO:teuthology.orchestra.run.smithi191.stdout:(32/120): ceph-mgr-rook-18.0.0-7537.gbcbf278c.e 862 kB/s | 50 kB 00:00 2023-12-03T11:22:09.994 INFO:teuthology.orchestra.run.smithi191.stdout:(33/120): ceph-prometheus-alerts-18.0.0-7537.gb 258 kB/s | 15 kB 00:00 2023-12-03T11:22:10.061 INFO:teuthology.orchestra.run.smithi191.stdout:(34/120): ceph-volume-18.0.0-7537.gbcbf278c.el9 2.8 MB/s | 262 kB 00:00 2023-12-03T11:22:10.138 INFO:teuthology.orchestra.run.smithi012.stdout:(7/120): ceph-mon-18.0.0-7537.gbcbf278c.el9.x86 5.5 MB/s | 4.4 MB 00:00 2023-12-03T11:22:10.161 INFO:teuthology.orchestra.run.smithi191.stdout:(35/120): cephadm-18.0.0-7537.gbcbf278c.el9.noa 4.5 MB/s | 774 kB 00:00 2023-12-03T11:22:10.212 INFO:teuthology.orchestra.run.smithi191.stdout:(36/120): flexiblas-3.0.4-8.el9.x86_64.rpm 650 kB/s | 32 kB 00:00 2023-12-03T11:22:10.237 INFO:teuthology.orchestra.run.smithi191.stdout:(37/120): boost-program-options-1.75.0-8.el9.x8 603 kB/s | 106 kB 00:00 2023-12-03T11:22:10.304 INFO:teuthology.orchestra.run.smithi191.stdout:(38/120): flexiblas-openblas-openmp-3.0.4-8.el9 259 kB/s | 17 kB 00:00 2023-12-03T11:22:10.385 INFO:teuthology.orchestra.run.smithi191.stdout:(39/120): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2023-12-03T11:22:10.413 INFO:teuthology.orchestra.run.smithi191.stdout:(40/120): jq-1.6-16.el9.x86_64.rpm 1.7 MB/s | 186 kB 00:00 2023-12-03T11:22:10.438 INFO:teuthology.orchestra.run.smithi191.stdout:(41/120): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.0 MB/s | 159 kB 00:00 2023-12-03T11:22:10.463 INFO:teuthology.orchestra.run.smithi191.stdout:(42/120): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2023-12-03T11:22:10.488 INFO:teuthology.orchestra.run.smithi191.stdout:(43/120): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-03T11:22:10.522 INFO:teuthology.orchestra.run.smithi191.stdout:(44/120): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2023-12-03T11:22:10.547 INFO:teuthology.orchestra.run.smithi191.stdout:(45/120): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-12-03T11:22:10.572 INFO:teuthology.orchestra.run.smithi191.stdout:(46/120): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-03T11:22:10.598 INFO:teuthology.orchestra.run.smithi191.stdout:(47/120): openblas-0.3.21-2.el9.x86_64.rpm 686 kB/s | 34 kB 00:00 2023-12-03T11:22:10.706 INFO:teuthology.orchestra.run.smithi191.stdout:(48/120): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2023-12-03T11:22:10.799 INFO:teuthology.orchestra.run.smithi191.stdout:(49/120): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 5.8 MB 00:00 2023-12-03T11:22:10.825 INFO:teuthology.orchestra.run.smithi191.stdout:(50/120): python3-jinja2-2.11.3-4.el9.noarch.rp 2.0 MB/s | 228 kB 00:00 2023-12-03T11:22:10.841 INFO:teuthology.orchestra.run.smithi191.stdout:(51/120): python3-jsonpatch-1.21-16.el9.noarch. 592 kB/s | 24 kB 00:00 2023-12-03T11:22:10.866 INFO:teuthology.orchestra.run.smithi191.stdout:(52/120): python3-jsonpointer-2.0-4.el9.noarch. 433 kB/s | 18 kB 00:00 2023-12-03T11:22:10.891 INFO:teuthology.orchestra.run.smithi191.stdout:(53/120): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-03T11:22:10.917 INFO:teuthology.orchestra.run.smithi191.stdout:(54/120): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2023-12-03T11:22:11.025 INFO:teuthology.orchestra.run.smithi191.stdout:(55/120): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2023-12-03T11:22:11.051 INFO:teuthology.orchestra.run.smithi191.stdout:(56/120): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2023-12-03T11:22:11.076 INFO:teuthology.orchestra.run.smithi191.stdout:(57/120): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2023-12-03T11:22:11.101 INFO:teuthology.orchestra.run.smithi191.stdout:(58/120): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2023-12-03T11:22:11.126 INFO:teuthology.orchestra.run.smithi191.stdout:(59/120): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-12-03T11:22:11.152 INFO:teuthology.orchestra.run.smithi191.stdout:(60/120): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-03T11:22:11.177 INFO:teuthology.orchestra.run.smithi191.stdout:(61/120): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2023-12-03T11:22:11.202 INFO:teuthology.orchestra.run.smithi191.stdout:(62/120): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2023-12-03T11:22:11.361 INFO:teuthology.orchestra.run.smithi191.stdout:(63/120): python3-toml-0.10.2-6.el9.noarch.rpm 247 kB/s | 39 kB 00:00 2023-12-03T11:22:11.536 INFO:teuthology.orchestra.run.smithi191.stdout:(64/120): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2023-12-03T11:22:11.561 INFO:teuthology.orchestra.run.smithi191.stdout:(65/120): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2023-12-03T11:22:11.587 INFO:teuthology.orchestra.run.smithi191.stdout:(66/120): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-12-03T11:22:11.612 INFO:teuthology.orchestra.run.smithi191.stdout:(67/120): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2023-12-03T11:22:11.637 INFO:teuthology.orchestra.run.smithi191.stdout:(68/120): libgfortran-11.4.1-2.3.el9.x86_64.rpm 16 MB/s | 803 kB 00:00 2023-12-03T11:22:11.662 INFO:teuthology.orchestra.run.smithi191.stdout:(69/120): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.9 MB/s | 195 kB 00:00 2023-12-03T11:22:11.688 INFO:teuthology.orchestra.run.smithi191.stdout:(70/120): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-03T11:22:11.721 INFO:teuthology.orchestra.run.smithi191.stdout:(71/120): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2023-12-03T11:22:11.730 INFO:teuthology.orchestra.run.smithi012.stdout:(8/120): ceph-radosgw-18.0.0-7537.gbcbf278c.el9 5.4 MB/s | 8.6 MB 00:01 2023-12-03T11:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout:(72/120): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2023-12-03T11:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout:(73/120): python3-pycparser-2.20-6.el9.noarch.r 2.9 MB/s | 124 kB 00:00 2023-12-03T11:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout:(74/120): python3-requests-2.25.1-7.el9.noarch. 2.2 MB/s | 113 kB 00:00 2023-12-03T11:22:11.797 INFO:teuthology.orchestra.run.smithi012.stdout:(9/120): ceph-selinux-18.0.0-7537.gbcbf278c.el9 363 kB/s | 24 kB 00:00 2023-12-03T11:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout:(75/120): python3-urllib3-1.26.5-4.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2023-12-03T11:22:11.907 INFO:teuthology.orchestra.run.smithi191.stdout:(76/120): ceph-mgr-diskprediction-local-18.0.0- 2.8 MB/s | 7.4 MB 00:02 2023-12-03T11:22:11.965 INFO:teuthology.orchestra.run.smithi191.stdout:(77/120): lua-devel-5.4.4-4.el9.x86_64.rpm 148 kB/s | 22 kB 00:00 2023-12-03T11:22:11.999 INFO:teuthology.orchestra.run.smithi191.stdout:(78/120): gperftools-libs-2.9.1-2.el9.x86_64.rp 3.3 MB/s | 309 kB 00:00 2023-12-03T11:22:12.024 INFO:teuthology.orchestra.run.smithi191.stdout:(79/120): libarrow-doc-9.0.0-9.el9.noarch.rpm 976 kB/s | 24 kB 00:00 2023-12-03T11:22:12.049 INFO:teuthology.orchestra.run.smithi191.stdout:(80/120): python3-devel-3.9.17-2.el9.x86_64.rpm 944 kB/s | 245 kB 00:00 2023-12-03T11:22:12.074 INFO:teuthology.orchestra.run.smithi191.stdout:(81/120): liboath-2.6.7-2.el9.x86_64.rpm 965 kB/s | 49 kB 00:00 2023-12-03T11:22:12.116 INFO:teuthology.orchestra.run.smithi191.stdout:(82/120): luarocks-3.9.2-1.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2023-12-03T11:22:12.142 INFO:teuthology.orchestra.run.smithi191.stdout:(83/120): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2023-12-03T11:22:12.175 INFO:teuthology.orchestra.run.smithi191.stdout:(84/120): parquet-libs-9.0.0-9.el9.x86_64.rpm 14 MB/s | 837 kB 00:00 2023-12-03T11:22:12.234 INFO:teuthology.orchestra.run.smithi191.stdout:(85/120): libarrow-9.0.0-9.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2023-12-03T11:22:12.260 INFO:teuthology.orchestra.run.smithi191.stdout:(86/120): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 517 kB/s | 43 kB 00:00 2023-12-03T11:22:12.285 INFO:teuthology.orchestra.run.smithi191.stdout:(87/120): python3-cachetools-4.2.4-1.el9.noarch 640 kB/s | 32 kB 00:00 2023-12-03T11:22:12.310 INFO:teuthology.orchestra.run.smithi191.stdout:(88/120): python3-certifi-2021.10.8-2.el9.noarc 299 kB/s | 15 kB 00:00 2023-12-03T11:22:12.335 INFO:teuthology.orchestra.run.smithi191.stdout:(89/120): python3-cheroot-8.6.0-4.el9.noarch.rp 3.3 MB/s | 172 kB 00:00 2023-12-03T11:22:12.360 INFO:teuthology.orchestra.run.smithi191.stdout:(90/120): python3-cherrypy-18.6.1-2.el9.noarch. 7.0 MB/s | 358 kB 00:00 2023-12-03T11:22:12.386 INFO:teuthology.orchestra.run.smithi191.stdout:(91/120): python3-asyncssh-2.13.2-3.el9.noarch. 2.2 MB/s | 548 kB 00:00 2023-12-03T11:22:12.411 INFO:teuthology.orchestra.run.smithi191.stdout:(92/120): python3-jaraco-8.2.1-3.el9.noarch.rpm 212 kB/s | 11 kB 00:00 2023-12-03T11:22:12.437 INFO:teuthology.orchestra.run.smithi191.stdout:(93/120): python3-jaraco-classes-3.2.1-5.el9.no 351 kB/s | 18 kB 00:00 2023-12-03T11:22:12.462 INFO:teuthology.orchestra.run.smithi191.stdout:(94/120): python3-jaraco-collections-3.0.0-8.el 460 kB/s | 23 kB 00:00 2023-12-03T11:22:12.487 INFO:teuthology.orchestra.run.smithi191.stdout:(95/120): python3-jaraco-functools-3.5.0-2.el9. 388 kB/s | 19 kB 00:00 2023-12-03T11:22:12.512 INFO:teuthology.orchestra.run.smithi191.stdout:(96/120): python3-jaraco-text-3.2.0-6.el9.noarc 393 kB/s | 20 kB 00:00 2023-12-03T11:22:12.538 INFO:teuthology.orchestra.run.smithi191.stdout:(97/120): python3-jwt+crypto-2.4.0-1.el9.noarch 180 kB/s | 9.0 kB 00:00 2023-12-03T11:22:12.563 INFO:teuthology.orchestra.run.smithi191.stdout:(98/120): python3-google-auth-2.23.4-1.el9.noar 870 kB/s | 197 kB 00:00 2023-12-03T11:22:12.588 INFO:teuthology.orchestra.run.smithi191.stdout:(99/120): python3-jwt-2.4.0-1.el9.noarch.rpm 543 kB/s | 41 kB 00:00 2023-12-03T11:22:12.613 INFO:teuthology.orchestra.run.smithi191.stdout:(100/120): python3-logutils-0.3.5-21.el9.noarch 925 kB/s | 46 kB 00:00 2023-12-03T11:22:12.638 INFO:teuthology.orchestra.run.smithi191.stdout:(101/120): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2023-12-03T11:22:12.680 INFO:teuthology.orchestra.run.smithi191.stdout:(102/120): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2023-12-03T11:22:12.706 INFO:teuthology.orchestra.run.smithi191.stdout:(103/120): python3-natsort-7.1.1-5.el9.noarch.r 625 kB/s | 58 kB 00:00 2023-12-03T11:22:12.731 INFO:teuthology.orchestra.run.smithi191.stdout:(104/120): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2023-12-03T11:22:12.756 INFO:teuthology.orchestra.run.smithi191.stdout:(105/120): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2023-12-03T11:22:12.782 INFO:teuthology.orchestra.run.smithi191.stdout:(106/120): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2023-12-03T11:22:12.807 INFO:teuthology.orchestra.run.smithi191.stdout:(107/120): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2023-12-03T11:22:12.832 INFO:teuthology.orchestra.run.smithi191.stdout:(108/120): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2023-12-03T11:22:12.857 INFO:teuthology.orchestra.run.smithi191.stdout:(109/120): python3-tempora-5.0.0-2.el9.noarch.r 712 kB/s | 36 kB 00:00 2023-12-03T11:22:12.883 INFO:teuthology.orchestra.run.smithi191.stdout:(110/120): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2023-12-03T11:22:12.908 INFO:teuthology.orchestra.run.smithi191.stdout:(111/120): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2023-12-03T11:22:12.925 INFO:teuthology.orchestra.run.smithi191.stdout:(112/120): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2023-12-03T11:22:12.950 INFO:teuthology.orchestra.run.smithi191.stdout:(113/120): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2023-12-03T11:22:12.975 INFO:teuthology.orchestra.run.smithi191.stdout:(114/120): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2023-12-03T11:22:13.001 INFO:teuthology.orchestra.run.smithi191.stdout:(115/120): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2023-12-03T11:22:13.026 INFO:teuthology.orchestra.run.smithi191.stdout:(116/120): python3-werkzeug-2.0.3-3.el9.1.noarc 3.5 MB/s | 427 kB 00:00 2023-12-03T11:22:13.076 INFO:teuthology.orchestra.run.smithi191.stdout:(117/120): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2023-12-03T11:22:13.101 INFO:teuthology.orchestra.run.smithi191.stdout:(118/120): s3cmd-2.3.0-4.el9.noarch.rpm 1.5 MB/s | 197 kB 00:00 2023-12-03T11:22:14.118 INFO:teuthology.orchestra.run.smithi191.stdout:(119/120): librados2-18.0.0-7537.gbcbf278c.el9. 3.0 MB/s | 3.3 MB 00:01 2023-12-03T11:22:14.318 INFO:teuthology.orchestra.run.smithi191.stdout:(120/120): librbd1-18.0.0-7537.gbcbf278c.el9.x8 2.4 MB/s | 3.0 MB 00:01 2023-12-03T11:22:14.321 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:22:14.321 INFO:teuthology.orchestra.run.smithi191.stdout:Total 17 MB/s | 181 MB 00:10 2023-12-03T11:22:14.347 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-12-03T11:22:14.446 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-12-03T11:22:14.447 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-12-03T11:22:15.272 INFO:teuthology.orchestra.run.smithi012.stdout:(10/120): ceph-osd-18.0.0-7537.gbcbf278c.el9.x8 3.2 MB/s | 18 MB 00:05 2023-12-03T11:22:15.455 INFO:teuthology.orchestra.run.smithi012.stdout:(11/120): libcephfs-devel-18.0.0-7537.gbcbf278c 170 kB/s | 31 kB 00:00 2023-12-03T11:22:15.697 INFO:teuthology.orchestra.run.smithi012.stdout:(12/120): libcephfs2-18.0.0-7537.gbcbf278c.el9. 2.9 MB/s | 718 kB 00:00 2023-12-03T11:22:15.781 INFO:teuthology.orchestra.run.smithi012.stdout:(13/120): libcephsqlite-18.0.0-7537.gbcbf278c.e 1.9 MB/s | 164 kB 00:00 2023-12-03T11:22:15.873 INFO:teuthology.orchestra.run.smithi012.stdout:(14/120): librados-devel-18.0.0-7537.gbcbf278c. 1.4 MB/s | 126 kB 00:00 2023-12-03T11:22:16.004 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-12-03T11:22:16.005 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-12-03T11:22:16.065 INFO:teuthology.orchestra.run.smithi012.stdout:(15/120): libradosstriper1-18.0.0-7537.gbcbf278 2.4 MB/s | 475 kB 00:00 2023-12-03T11:22:16.990 INFO:teuthology.orchestra.run.smithi012.stdout:(16/120): ceph-common-18.0.0-7537.gbcbf278c.el9 2.0 MB/s | 19 MB 00:09 2023-12-03T11:22:17.090 INFO:teuthology.orchestra.run.smithi012.stdout:(17/120): python3-ceph-argparse-18.0.0-7537.gbc 453 kB/s | 45 kB 00:00 2023-12-03T11:22:17.198 INFO:teuthology.orchestra.run.smithi012.stdout:(18/120): python3-ceph-common-18.0.0-7537.gbcbf 1.2 MB/s | 129 kB 00:00 2023-12-03T11:22:17.315 INFO:teuthology.orchestra.run.smithi012.stdout:(19/120): python3-cephfs-18.0.0-7537.gbcbf278c. 1.4 MB/s | 162 kB 00:00 2023-12-03T11:22:17.507 INFO:teuthology.orchestra.run.smithi012.stdout:(20/120): python3-rados-18.0.0-7537.gbcbf278c.e 1.6 MB/s | 320 kB 00:00 2023-12-03T11:22:17.708 INFO:teuthology.orchestra.run.smithi012.stdout:(21/120): python3-rbd-18.0.0-7537.gbcbf278c.el9 1.5 MB/s | 298 kB 00:00 2023-12-03T11:22:17.783 INFO:teuthology.orchestra.run.smithi012.stdout:(22/120): librgw2-18.0.0-7537.gbcbf278c.el9.x86 2.8 MB/s | 4.8 MB 00:01 2023-12-03T11:22:17.808 INFO:teuthology.orchestra.run.smithi012.stdout:(23/120): python3-rgw-18.0.0-7537.gbcbf278c.el9 992 kB/s | 99 kB 00:00 2023-12-03T11:22:17.883 INFO:teuthology.orchestra.run.smithi012.stdout:(24/120): rbd-fuse-18.0.0-7537.gbcbf278c.el9.x8 854 kB/s | 85 kB 00:00 2023-12-03T11:22:17.984 INFO:teuthology.orchestra.run.smithi012.stdout:(25/120): rbd-nbd-18.0.0-7537.gbcbf278c.el9.x86 1.7 MB/s | 170 kB 00:00 2023-12-03T11:22:18.092 INFO:teuthology.orchestra.run.smithi012.stdout:(26/120): ceph-grafana-dashboards-18.0.0-7537.g 258 kB/s | 28 kB 00:00 2023-12-03T11:22:18.209 INFO:teuthology.orchestra.run.smithi012.stdout:(27/120): ceph-mgr-cephadm-18.0.0-7537.gbcbf278 1.1 MB/s | 133 kB 00:00 2023-12-03T11:22:18.776 INFO:teuthology.orchestra.run.smithi012.stdout:(28/120): ceph-mgr-dashboard-18.0.0-7537.gbcbf2 3.1 MB/s | 1.8 MB 00:00 2023-12-03T11:22:19.576 INFO:teuthology.orchestra.run.smithi012.stdout:(29/120): rbd-mirror-18.0.0-7537.gbcbf278c.el9. 1.7 MB/s | 3.0 MB 00:01 2023-12-03T11:22:19.760 INFO:teuthology.orchestra.run.smithi012.stdout:(30/120): ceph-mgr-modules-core-18.0.0-7537.gbc 1.3 MB/s | 243 kB 00:00 2023-12-03T11:22:19.860 INFO:teuthology.orchestra.run.smithi012.stdout:(31/120): ceph-mgr-rook-18.0.0-7537.gbcbf278c.e 502 kB/s | 50 kB 00:00 2023-12-03T11:22:19.952 INFO:teuthology.orchestra.run.smithi012.stdout:(32/120): ceph-prometheus-alerts-18.0.0-7537.gb 165 kB/s | 15 kB 00:00 2023-12-03T11:22:20.136 INFO:teuthology.orchestra.run.smithi012.stdout:(33/120): ceph-volume-18.0.0-7537.gbcbf278c.el9 1.4 MB/s | 262 kB 00:00 2023-12-03T11:22:20.611 INFO:teuthology.orchestra.run.smithi012.stdout:(34/120): cephadm-18.0.0-7537.gbcbf278c.el9.noa 1.6 MB/s | 774 kB 00:00 2023-12-03T11:22:20.640 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-12-03T11:22:20.704 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/122 2023-12-03T11:22:20.720 INFO:teuthology.orchestra.run.smithi012.stdout:(35/120): boost-program-options-1.75.0-8.el9.x8 979 kB/s | 106 kB 00:00 2023-12-03T11:22:20.738 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/122 2023-12-03T11:22:20.753 INFO:teuthology.orchestra.run.smithi012.stdout:(36/120): flexiblas-3.0.4-8.el9.x86_64.rpm 954 kB/s | 32 kB 00:00 2023-12-03T11:22:20.791 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/122 2023-12-03T11:22:20.831 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/122 2023-12-03T11:22:20.870 INFO:teuthology.orchestra.run.smithi012.stdout:(37/120): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2023-12-03T11:22:20.885 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/122 2023-12-03T11:22:20.904 INFO:teuthology.orchestra.run.smithi012.stdout:(38/120): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2023-12-03T11:22:20.953 INFO:teuthology.orchestra.run.smithi012.stdout:(39/120): jq-1.6-16.el9.x86_64.rpm 3.8 MB/s | 186 kB 00:00 2023-12-03T11:22:20.988 INFO:teuthology.orchestra.run.smithi012.stdout:(40/120): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.5 MB/s | 159 kB 00:00 2023-12-03T11:22:21.021 INFO:teuthology.orchestra.run.smithi012.stdout:(41/120): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2023-12-03T11:22:21.064 INFO:teuthology.orchestra.run.smithi012.stdout:(42/120): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2023-12-03T11:22:21.164 INFO:teuthology.orchestra.run.smithi012.stdout:(43/120): ceph-mgr-diskprediction-local-18.0.0- 3.1 MB/s | 7.4 MB 00:02 2023-12-03T11:22:21.189 INFO:teuthology.orchestra.run.smithi012.stdout:(44/120): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.2 MB/s | 282 kB 00:00 2023-12-03T11:22:21.215 INFO:teuthology.orchestra.run.smithi012.stdout:(45/120): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-03T11:22:21.217 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 6/122 2023-12-03T11:22:21.240 INFO:teuthology.orchestra.run.smithi012.stdout:(46/120): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2023-12-03T11:22:21.257 INFO:teuthology.orchestra.run.smithi012.stdout:(47/120): openblas-0.3.21-2.el9.x86_64.rpm 828 kB/s | 34 kB 00:00 2023-12-03T11:22:21.264 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:22:21.365 INFO:teuthology.orchestra.run.smithi012.stdout:(48/120): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2023-12-03T11:22:21.375 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:22:21.432 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:22:21.474 INFO:teuthology.orchestra.run.smithi012.stdout:(49/120): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2023-12-03T11:22:21.499 INFO:teuthology.orchestra.run.smithi012.stdout:(50/120): python3-jinja2-2.11.3-4.el9.noarch.rp 1.7 MB/s | 228 kB 00:00 2023-12-03T11:22:21.516 INFO:teuthology.orchestra.run.smithi012.stdout:(51/120): python3-jsonpatch-1.21-16.el9.noarch. 590 kB/s | 24 kB 00:00 2023-12-03T11:22:21.533 INFO:teuthology.orchestra.run.smithi012.stdout:(52/120): python3-jsonpointer-2.0-4.el9.noarch. 539 kB/s | 18 kB 00:00 2023-12-03T11:22:21.558 INFO:teuthology.orchestra.run.smithi012.stdout:(53/120): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2023-12-03T11:22:21.579 INFO:teuthology.orchestra.run.smithi012.stdout:(54/120): python3-markupsafe-1.1.1-12.el9.x86_6 704 kB/s | 32 kB 00:00 2023-12-03T11:22:21.700 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:22:21.748 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:22:21.763 INFO:teuthology.orchestra.run.smithi012.stdout:(55/120): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 4.8 MB 00:00 2023-12-03T11:22:21.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:22:21.788 INFO:teuthology.orchestra.run.smithi012.stdout:(56/120): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 217 kB 00:00 2023-12-03T11:22:21.813 INFO:teuthology.orchestra.run.smithi012.stdout:(57/120): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2023-12-03T11:22:21.839 INFO:teuthology.orchestra.run.smithi012.stdout:(58/120): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2023-12-03T11:22:21.864 INFO:teuthology.orchestra.run.smithi012.stdout:(59/120): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-12-03T11:22:21.877 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:22:21.889 INFO:teuthology.orchestra.run.smithi012.stdout:(60/120): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-03T11:22:21.901 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:22:21.914 INFO:teuthology.orchestra.run.smithi012.stdout:(61/120): python3-pytz-2021.1-5.el9.noarch.rpm 946 kB/s | 47 kB 00:00 2023-12-03T11:22:21.940 INFO:teuthology.orchestra.run.smithi012.stdout:(62/120): python3-requests-oauthlib-1.3.0-12.el 853 kB/s | 43 kB 00:00 2023-12-03T11:22:22.014 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:22:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 12/122 2023-12-03T11:22:22.117 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 13/122 2023-12-03T11:22:22.171 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 14/122 2023-12-03T11:22:22.209 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 15/122 2023-12-03T11:22:22.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 16/122 2023-12-03T11:22:22.276 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 17/122 2023-12-03T11:22:22.323 INFO:teuthology.orchestra.run.smithi012.stdout:(63/120): python3-toml-0.10.2-6.el9.noarch.rpm 101 kB/s | 39 kB 00:00 2023-12-03T11:22:22.333 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/122 2023-12-03T11:22:22.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/122 2023-12-03T11:22:22.401 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/122 2023-12-03T11:22:22.428 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-5.4.4-4.el9.x86_64 21/122 2023-12-03T11:22:22.482 INFO:teuthology.orchestra.run.smithi012.stdout:(64/120): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2023-12-03T11:22:22.507 INFO:teuthology.orchestra.run.smithi012.stdout:(65/120): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2023-12-03T11:22:22.508 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/122 2023-12-03T11:22:22.524 INFO:teuthology.orchestra.run.smithi012.stdout:(66/120): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-03T11:22:22.549 INFO:teuthology.orchestra.run.smithi012.stdout:(67/120): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2023-12-03T11:22:22.626 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 23/122 2023-12-03T11:22:22.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 24/122 2023-12-03T11:22:22.822 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 25/122 2023-12-03T11:22:22.893 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 26/122 2023-12-03T11:22:22.933 INFO:teuthology.orchestra.run.smithi012.stdout:(68/120): ceph-test-18.0.0-7537.gbcbf278c.el9.x 3.7 MB/s | 41 MB 00:11 2023-12-03T11:22:22.958 INFO:teuthology.orchestra.run.smithi012.stdout:(69/120): libgfortran-11.4.1-2.3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2023-12-03T11:22:22.968 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : luarocks-3.9.2-1.el9.noarch 27/122 2023-12-03T11:22:22.983 INFO:teuthology.orchestra.run.smithi012.stdout:(70/120): libquadmath-11.4.1-2.3.el9.x86_64.rpm 451 kB/s | 195 kB 00:00 2023-12-03T11:22:23.008 INFO:teuthology.orchestra.run.smithi012.stdout:(71/120): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2023-12-03T11:22:23.059 INFO:teuthology.orchestra.run.smithi012.stdout:(72/120): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2023-12-03T11:22:23.084 INFO:teuthology.orchestra.run.smithi012.stdout:(73/120): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2023-12-03T11:22:23.104 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 28/122 2023-12-03T11:22:23.109 INFO:teuthology.orchestra.run.smithi012.stdout:(74/120): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2023-12-03T11:22:23.135 INFO:teuthology.orchestra.run.smithi012.stdout:(75/120): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-12-03T11:22:23.155 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 29/122 2023-12-03T11:22:23.160 INFO:teuthology.orchestra.run.smithi012.stdout:(76/120): python3-urllib3-1.26.5-4.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2023-12-03T11:22:23.235 INFO:teuthology.orchestra.run.smithi012.stdout:(77/120): lua-devel-5.4.4-4.el9.x86_64.rpm 224 kB/s | 22 kB 00:00 2023-12-03T11:22:23.360 INFO:teuthology.orchestra.run.smithi012.stdout:(78/120): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2023-12-03T11:22:23.690 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 30/122 2023-12-03T11:22:23.719 INFO:teuthology.orchestra.run.smithi012.stdout:(79/120): libarrow-doc-9.0.0-9.el9.noarch.rpm 68 kB/s | 24 kB 00:00 2023-12-03T11:22:23.768 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 31/122 2023-12-03T11:22:23.769 INFO:teuthology.orchestra.run.smithi012.stdout:(80/120): gperftools-libs-2.9.1-2.el9.x86_64.rp 507 kB/s | 309 kB 00:00 2023-12-03T11:22:23.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 32/122 2023-12-03T11:22:23.843 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 33/122 2023-12-03T11:22:23.865 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 34/122 2023-12-03T11:22:23.869 INFO:teuthology.orchestra.run.smithi012.stdout:(81/120): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2023-12-03T11:22:23.886 INFO:teuthology.orchestra.run.smithi012.stdout:(82/120): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2023-12-03T11:22:23.970 INFO:teuthology.orchestra.run.smithi012.stdout:(83/120): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2023-12-03T11:22:23.972 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : re2-1:20211101-3.el9.x86_64 35/122 2023-12-03T11:22:24.010 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 36/122 2023-12-03T11:22:24.066 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 37/122 2023-12-03T11:22:24.142 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 38/122 2023-12-03T11:22:24.186 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 39/122 2023-12-03T11:22:24.223 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 40/122 2023-12-03T11:22:24.223 INFO:teuthology.orchestra.run.smithi012.stdout:(84/120): libarrow-9.0.0-9.el9.x86_64.rpm 4.5 MB/s | 4.4 MB 00:00 2023-12-03T11:22:24.265 INFO:teuthology.orchestra.run.smithi012.stdout:(85/120): python3-asyncssh-2.13.2-3.el9.noarch. 1.8 MB/s | 548 kB 00:00 2023-12-03T11:22:24.270 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 41/122 2023-12-03T11:22:24.290 INFO:teuthology.orchestra.run.smithi012.stdout:(86/120): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.0 MB/s | 837 kB 00:00 2023-12-03T11:22:24.314 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 42/122 2023-12-03T11:22:24.315 INFO:teuthology.orchestra.run.smithi012.stdout:(87/120): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2023-12-03T11:22:24.356 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 43/122 2023-12-03T11:22:24.357 INFO:teuthology.orchestra.run.smithi012.stdout:(88/120): python3-cachetools-4.2.4-1.el9.noarch 350 kB/s | 32 kB 00:00 2023-12-03T11:22:24.382 INFO:teuthology.orchestra.run.smithi012.stdout:(89/120): python3-certifi-2021.10.8-2.el9.noarc 163 kB/s | 15 kB 00:00 2023-12-03T11:22:24.416 INFO:teuthology.orchestra.run.smithi012.stdout:(90/120): python3-cheroot-8.6.0-4.el9.noarch.rp 1.7 MB/s | 172 kB 00:00 2023-12-03T11:22:24.466 INFO:teuthology.orchestra.run.smithi012.stdout:(91/120): python3-cherrypy-18.6.1-2.el9.noarch. 3.2 MB/s | 358 kB 00:00 2023-12-03T11:22:24.483 INFO:teuthology.orchestra.run.smithi012.stdout:(92/120): python3-google-auth-2.23.4-1.el9.noar 1.9 MB/s | 197 kB 00:00 2023-12-03T11:22:24.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 44/122 2023-12-03T11:22:24.508 INFO:teuthology.orchestra.run.smithi012.stdout:(93/120): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2023-12-03T11:22:24.542 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 45/122 2023-12-03T11:22:24.558 INFO:teuthology.orchestra.run.smithi012.stdout:(94/120): python3-jaraco-classes-3.2.1-5.el9.no 191 kB/s | 18 kB 00:00 2023-12-03T11:22:24.575 INFO:teuthology.orchestra.run.smithi012.stdout:(95/120): python3-jaraco-collections-3.0.0-8.el 251 kB/s | 23 kB 00:00 2023-12-03T11:22:24.584 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 46/122 2023-12-03T11:22:24.600 INFO:teuthology.orchestra.run.smithi012.stdout:(96/120): python3-jaraco-functools-3.5.0-2.el9. 211 kB/s | 19 kB 00:00 2023-12-03T11:22:24.651 INFO:teuthology.orchestra.run.smithi012.stdout:(97/120): python3-jaraco-text-3.2.0-6.el9.noarc 214 kB/s | 20 kB 00:00 2023-12-03T11:22:24.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 47/122 2023-12-03T11:22:24.701 INFO:teuthology.orchestra.run.smithi012.stdout:(98/120): python3-jwt+crypto-2.4.0-1.el9.noarch 72 kB/s | 9.0 kB 00:00 2023-12-03T11:22:24.709 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 48/122 2023-12-03T11:22:24.718 INFO:teuthology.orchestra.run.smithi012.stdout:(99/120): python3-jwt-2.4.0-1.el9.noarch.rpm 349 kB/s | 41 kB 00:00 2023-12-03T11:22:24.749 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 49/122 2023-12-03T11:22:24.782 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 50/122 2023-12-03T11:22:24.802 INFO:teuthology.orchestra.run.smithi012.stdout:(100/120): python3-logutils-0.3.5-21.el9.noarch 461 kB/s | 46 kB 00:00 2023-12-03T11:22:24.827 INFO:teuthology.orchestra.run.smithi012.stdout:(101/120): python3-more-itertools-8.12.0-2.el9. 727 kB/s | 79 kB 00:00 2023-12-03T11:22:24.860 INFO:teuthology.orchestra.run.smithi012.stdout:(102/120): python3-kubernetes-26.1.0-2.el9.noar 4.9 MB/s | 1.0 MB 00:00 2023-12-03T11:22:24.880 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 51/122 2023-12-03T11:22:24.918 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 52/122 2023-12-03T11:22:24.919 INFO:teuthology.orchestra.run.smithi012.stdout:(103/120): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2023-12-03T11:22:24.944 INFO:teuthology.orchestra.run.smithi012.stdout:(104/120): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2023-12-03T11:22:24.975 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 53/122 2023-12-03T11:22:24.978 INFO:teuthology.orchestra.run.smithi012.stdout:(105/120): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2023-12-03T11:22:25.020 INFO:teuthology.orchestra.run.smithi012.stdout:(106/120): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2023-12-03T11:22:25.037 INFO:teuthology.orchestra.run.smithi012.stdout:(107/120): python3-repoze-lru-0.7-16.el9.noarch 333 kB/s | 31 kB 00:00 2023-12-03T11:22:25.039 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 54/122 2023-12-03T11:22:25.087 INFO:teuthology.orchestra.run.smithi012.stdout:(108/120): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2023-12-03T11:22:25.112 INFO:teuthology.orchestra.run.smithi012.stdout:(109/120): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2023-12-03T11:22:25.138 INFO:teuthology.orchestra.run.smithi012.stdout:(110/120): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2023-12-03T11:22:25.213 INFO:teuthology.orchestra.run.smithi012.stdout:(111/120): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2023-12-03T11:22:25.232 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 55/122 2023-12-03T11:22:25.238 INFO:teuthology.orchestra.run.smithi012.stdout:(112/120): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2023-12-03T11:22:25.263 INFO:teuthology.orchestra.run.smithi012.stdout:(113/120): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2023-12-03T11:22:25.280 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 56/122 2023-12-03T11:22:25.322 INFO:teuthology.orchestra.run.smithi012.stdout:(114/120): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2023-12-03T11:22:25.347 INFO:teuthology.orchestra.run.smithi012.stdout:(115/120): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2023-12-03T11:22:25.372 INFO:teuthology.orchestra.run.smithi012.stdout:(116/120): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2023-12-03T11:22:25.423 INFO:teuthology.orchestra.run.smithi012.stdout:(117/120): s3cmd-2.3.0-4.el9.noarch.rpm 1.9 MB/s | 197 kB 00:00 2023-12-03T11:22:25.428 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 57/122 2023-12-03T11:22:25.475 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 58/122 2023-12-03T11:22:25.503 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 59/122 2023-12-03T11:22:25.538 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 60/122 2023-12-03T11:22:25.570 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 61/122 2023-12-03T11:22:25.602 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-15.el9.x86_64 62/122 2023-12-03T11:22:25.607 INFO:teuthology.orchestra.run.smithi012.stdout:(118/120): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2023-12-03T11:22:25.647 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 63/122 2023-12-03T11:22:25.682 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 64/122 2023-12-03T11:22:25.712 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:22:25.789 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 66/122 2023-12-03T11:22:25.822 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 67/122 2023-12-03T11:22:25.848 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 68/122 2023-12-03T11:22:25.889 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 69/122 2023-12-03T11:22:25.923 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 70/122 2023-12-03T11:22:26.014 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 71/122 2023-12-03T11:22:26.051 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 72/122 2023-12-03T11:22:26.157 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 73/122 2023-12-03T11:22:26.219 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 74/122 2023-12-03T11:22:26.407 INFO:teuthology.orchestra.run.smithi012.stdout:(119/120): librados2-18.0.0-7537.gbcbf278c.el9. 3.2 MB/s | 3.3 MB 00:01 2023-12-03T11:22:26.623 INFO:teuthology.orchestra.run.smithi012.stdout:(120/120): librbd1-18.0.0-7537.gbcbf278c.el9.x8 2.5 MB/s | 3.0 MB 00:01 2023-12-03T11:22:26.629 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:22:26.629 INFO:teuthology.orchestra.run.smithi012.stdout:Total 9.2 MB/s | 181 MB 00:19 2023-12-03T11:22:26.654 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-03T11:22:26.762 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-03T11:22:26.762 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-03T11:22:26.964 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/122 2023-12-03T11:22:27.007 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 76/122 2023-12-03T11:22:27.049 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 77/122 2023-12-03T11:22:27.081 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 78/122 2023-12-03T11:22:27.252 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/122 2023-12-03T11:22:27.288 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/122 2023-12-03T11:22:27.366 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/122 2023-12-03T11:22:28.063 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/122 2023-12-03T11:22:28.178 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/122 2023-12-03T11:22:28.340 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-03T11:22:28.341 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-03T11:22:29.453 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/122 2023-12-03T11:22:29.538 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/122 2023-12-03T11:22:29.569 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 86/122 2023-12-03T11:22:29.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-16.el9.x86_64 87/122 2023-12-03T11:22:29.898 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 88/122 2023-12-03T11:22:29.932 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:22:30.022 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:22:30.045 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 90/122 2023-12-03T11:22:30.073 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 91/122 2023-12-03T11:22:30.500 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 92/122 2023-12-03T11:22:30.526 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:22:30.581 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:22:30.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 94/122 2023-12-03T11:22:32.718 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:32.759 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:33.233 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-03T11:22:33.309 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/122 2023-12-03T11:22:33.334 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/122 2023-12-03T11:22:33.334 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:33.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:22:33.379 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/122 2023-12-03T11:22:33.419 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/122 2023-12-03T11:22:33.455 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:22:33.455 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-03T11:22:33.455 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-03T11:22:33.456 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:33.471 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/122 2023-12-03T11:22:33.497 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:22:33.815 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 6/122 2023-12-03T11:22:33.869 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:22:33.991 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:22:34.020 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:22:34.112 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:22:34.158 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:22:34.188 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:22:34.275 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:22:34.297 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:22:34.411 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 11/122 2023-12-03T11:22:34.443 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 12/122 2023-12-03T11:22:34.507 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : liboath-2.6.7-2.el9.x86_64 13/122 2023-12-03T11:22:34.559 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 14/122 2023-12-03T11:22:34.587 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 15/122 2023-12-03T11:22:34.624 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 16/122 2023-12-03T11:22:34.664 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 17/122 2023-12-03T11:22:34.740 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 18/122 2023-12-03T11:22:34.780 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 19/122 2023-12-03T11:22:34.813 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 20/122 2023-12-03T11:22:34.841 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-5.4.4-4.el9.x86_64 21/122 2023-12-03T11:22:34.913 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/122 2023-12-03T11:22:34.946 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 23/122 2023-12-03T11:22:34.989 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 24/122 2023-12-03T11:22:35.018 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 25/122 2023-12-03T11:22:35.084 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 26/122 2023-12-03T11:22:35.167 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : luarocks-3.9.2-1.el9.noarch 27/122 2023-12-03T11:22:35.307 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 28/122 2023-12-03T11:22:35.360 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 29/122 2023-12-03T11:22:35.921 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rsa-4.9-2.el9.noarch 30/122 2023-12-03T11:22:36.007 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 31/122 2023-12-03T11:22:36.052 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 32/122 2023-12-03T11:22:36.089 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 33/122 2023-12-03T11:22:36.120 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 34/122 2023-12-03T11:22:36.230 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : re2-1:20211101-3.el9.x86_64 35/122 2023-12-03T11:22:36.265 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 36/122 2023-12-03T11:22:36.322 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 37/122 2023-12-03T11:22:36.395 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 38/122 2023-12-03T11:22:36.432 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 39/122 2023-12-03T11:22:36.461 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 40/122 2023-12-03T11:22:36.508 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 41/122 2023-12-03T11:22:36.553 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 42/122 2023-12-03T11:22:36.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 43/122 2023-12-03T11:22:36.736 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 44/122 2023-12-03T11:22:36.780 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 45/122 2023-12-03T11:22:36.821 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 46/122 2023-12-03T11:22:36.918 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 47/122 2023-12-03T11:22:36.955 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 48/122 2023-12-03T11:22:36.996 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 49/122 2023-12-03T11:22:37.028 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 50/122 2023-12-03T11:22:37.128 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 51/122 2023-12-03T11:22:37.181 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 52/122 2023-12-03T11:22:37.239 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-ply-3.11-14.el9.noarch 53/122 2023-12-03T11:22:37.303 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 54/122 2023-12-03T11:22:37.516 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 55/122 2023-12-03T11:22:37.560 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 56/122 2023-12-03T11:22:37.706 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 57/122 2023-12-03T11:22:37.746 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 58/122 2023-12-03T11:22:37.766 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 59/122 2023-12-03T11:22:37.800 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 60/122 2023-12-03T11:22:37.852 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 61/122 2023-12-03T11:22:37.865 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fuse-2.9.9-15.el9.x86_64 62/122 2023-12-03T11:22:37.911 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 63/122 2023-12-03T11:22:37.944 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 64/122 2023-12-03T11:22:37.974 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:22:38.054 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 66/122 2023-12-03T11:22:38.085 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 67/122 2023-12-03T11:22:38.118 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 68/122 2023-12-03T11:22:38.161 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 69/122 2023-12-03T11:22:38.195 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 70/122 2023-12-03T11:22:38.289 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 71/122 2023-12-03T11:22:38.332 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 72/122 2023-12-03T11:22:38.440 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 73/122 2023-12-03T11:22:38.491 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 74/122 2023-12-03T11:22:39.265 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 75/122 2023-12-03T11:22:39.312 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 76/122 2023-12-03T11:22:39.346 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 77/122 2023-12-03T11:22:39.429 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 78/122 2023-12-03T11:22:39.750 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-0.3.21-2.el9.x86_64 79/122 2023-12-03T11:22:39.776 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 80/122 2023-12-03T11:22:39.849 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 81/122 2023-12-03T11:22:40.544 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 82/122 2023-12-03T11:22:40.745 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 83/122 2023-12-03T11:22:42.073 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/122 2023-12-03T11:22:42.143 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 85/122 2023-12-03T11:22:42.173 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 86/122 2023-12-03T11:22:42.207 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jq-1.6-16.el9.x86_64 87/122 2023-12-03T11:22:42.511 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 88/122 2023-12-03T11:22:42.536 INFO:teuthology.orchestra.run.smithi012.stdout: Upgrading : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:22:42.611 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 89/122 2023-12-03T11:22:42.641 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 90/122 2023-12-03T11:22:42.678 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 91/122 2023-12-03T11:22:43.130 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 92/122 2023-12-03T11:22:43.156 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:22:43.227 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 93/122 2023-12-03T11:22:43.249 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 94/122 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2023-12-03T11:22:45.290 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2023-12-03T11:22:45.291 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:45.382 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:45.412 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:45.458 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:22:45.476 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:22:45.476 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:45.476 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-03T11:22:45.476 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:45.981 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 95/122 2023-12-03T11:22:46.015 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:22:46.093 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 96/122 2023-12-03T11:22:46.093 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-03T11:22:46.093 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-03T11:22:46.093 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:22:46.133 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:22:47.036 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:22:47.062 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:47.253 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:22:47.276 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:47.721 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:22:47.743 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:22:47.744 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:47.744 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-03T11:22:47.744 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:22:47.744 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:22:47.744 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:48.192 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:22:48.236 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:22:48.266 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 103/122 2023-12-03T11:22:48.327 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 104/122 2023-12-03T11:22:48.353 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:22:48.572 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:22:48.602 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:22:49.564 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:22:49.594 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:22:49.693 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:22:49.868 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 108/122 2023-12-03T11:22:49.896 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:22:49.919 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:22:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:22:50.745 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 111/122 2023-12-03T11:22:50.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:22:50.796 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:50.849 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:22:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:22:50.871 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:50.871 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-03T11:22:50.871 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:51.166 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:22:51.188 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:22:51.188 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:51.188 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-03T11:22:51.188 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:22:51.189 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:22:51.189 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:22:55.003 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 115/122 2023-12-03T11:22:55.098 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 116/122 2023-12-03T11:22:55.201 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 117/122 2023-12-03T11:22:55.234 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 118/122 2023-12-03T11:22:55.285 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 119/122 2023-12-03T11:22:55.316 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : s3cmd-2.3.0-4.el9.noarch 120/122 2023-12-03T11:22:55.316 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:22:55.400 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:22:55.400 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 97/122 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /sys 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /proc 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /mnt 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /var/tmp 2023-12-03T11:22:58.167 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /home 2023-12-03T11:22:58.168 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /root 2023-12-03T11:22:58.168 INFO:teuthology.orchestra.run.smithi012.stdout:skipping the directory /tmp 2023-12-03T11:22:58.168 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:22:58.325 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:22:58.341 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 98/122 2023-12-03T11:22:58.341 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:58.341 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-03T11:22:58.341 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:22:59.962 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:22:59.988 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 99/122 2023-12-03T11:22:59.989 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:22:59.989 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-03T11:22:59.989 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:22:59.989 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-03T11:22:59.989 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 1/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 3/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 4/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 5/122 2023-12-03T11:23:00.018 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 6/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 11/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 12/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 13/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 14/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 15/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 16/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 17/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 18/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 19/122 2023-12-03T11:23:00.019 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 20/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 21/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 22/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 23/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 24/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 25/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 26/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 27/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 28/122 2023-12-03T11:23:00.020 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 29/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 30/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 31/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 32/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 33/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 34/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 35/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 36/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 37/122 2023-12-03T11:23:00.031 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 38/122 2023-12-03T11:23:00.032 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 39/122 2023-12-03T11:23:00.032 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 40/122 2023-12-03T11:23:00.032 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-16.el9.x86_64 41/122 2023-12-03T11:23:00.032 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 42/122 2023-12-03T11:23:00.032 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 43/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 44/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 45/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-5.4.4-4.el9.x86_64 46/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 47/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 48/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 49/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 50/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 51/122 2023-12-03T11:23:00.033 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 52/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 53/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 54/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 55/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 56/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 57/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 58/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 59/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 60/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 61/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 62/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 63/122 2023-12-03T11:23:00.034 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 64/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 66/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 67/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 68/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 69/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 70/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 71/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 72/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 73/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 74/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 75/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 76/122 2023-12-03T11:23:00.035 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 77/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 78/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 79/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 80/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 81/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 82/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 83/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 84/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 85/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 86/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 87/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 88/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 89/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 90/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 91/122 2023-12-03T11:23:00.036 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 92/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 93/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 94/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 95/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 96/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 97/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 98/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 99/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 100/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 101/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 102/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 103/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 104/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 105/122 2023-12-03T11:23:00.037 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 106/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 107/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 108/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 109/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 110/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 111/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 112/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 113/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 114/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 115/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : re2-1:20211101-3.el9.x86_64 116/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : s3cmd-2.3.0-4.el9.noarch 117/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 118/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 119/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 120/122 2023-12-03T11:23:00.038 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 121/122 2023-12-03T11:23:00.196 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 100/122 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-03T11:23:00.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:00.688 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 101/122 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-03T11:23:00.712 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:01.178 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:23:01.212 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 102/122 2023-12-03T11:23:01.242 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 103/122 2023-12-03T11:23:01.305 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 104/122 2023-12-03T11:23:01.335 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:23:01.568 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 105/122 2023-12-03T11:23:01.594 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.286 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.287 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-16.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.288 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: lua-5.4.4-4.el9.x86_64 2023-12-03T11:23:02.289 INFO:teuthology.orchestra.run.smithi191.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.290 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-03T11:23:02.291 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-03T11:23:02.292 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-03T11:23:02.293 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-03T11:23:02.294 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: re2-1:20211101-3.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: s3cmd-2.3.0-4.el9.noarch 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:02.295 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-12-03T11:23:02.602 DEBUG:teuthology.parallel:result is None 2023-12-03T11:23:02.609 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 106/122 2023-12-03T11:23:02.636 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:23:02.739 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 107/122 2023-12-03T11:23:02.901 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 108/122 2023-12-03T11:23:02.938 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:23:02.962 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 109/122 2023-12-03T11:23:02.963 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:02.963 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-03T11:23:02.963 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:23:02.963 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-03T11:23:02.963 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:03.014 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:23:03.025 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 110/122 2023-12-03T11:23:03.840 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 111/122 2023-12-03T11:23:03.873 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 112/122 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-03T11:23:03.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:03.949 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:23:03.972 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 113/122 2023-12-03T11:23:03.972 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:03.972 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-03T11:23:03.972 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:04.283 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:23:04.306 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 114/122 2023-12-03T11:23:04.307 INFO:teuthology.orchestra.run.smithi012.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-03T11:23:04.307 INFO:teuthology.orchestra.run.smithi012.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-03T11:23:04.307 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:23:04.307 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-03T11:23:04.307 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:08.336 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 115/122 2023-12-03T11:23:08.392 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 116/122 2023-12-03T11:23:08.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 117/122 2023-12-03T11:23:08.526 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 118/122 2023-12-03T11:23:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 119/122 2023-12-03T11:23:08.624 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : s3cmd-2.3.0-4.el9.noarch 120/122 2023-12-03T11:23:08.624 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:23:08.708 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 121/122 2023-12-03T11:23:08.708 INFO:teuthology.orchestra.run.smithi012.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:23:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:23:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 1/122 2023-12-03T11:23:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2/122 2023-12-03T11:23:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 3/122 2023-12-03T11:23:12.918 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 4/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7537.gbcbf2 5/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 6/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 7/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 8/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 9/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 10/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 11/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 12/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_ 13/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 14/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 15/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_6 16/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86 17/122 2023-12-03T11:23:12.919 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 18/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el 19/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9. 20/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_6 21/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 22/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 23/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 24/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 25/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 26/122 2023-12-03T11:23:12.920 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 27/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c. 28/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noa 29/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.n 30/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcb 31/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el 32/122 2023-12-03T11:23:12.929 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 33/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.e 34/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 35/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 36/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 37/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 38/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 39/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 40/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jq-1.6-16.el9.x86_64 41/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 42/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 43/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 44/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 45/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-5.4.4-4.el9.x86_64 46/122 2023-12-03T11:23:12.930 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 47/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 48/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 49/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 50/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 51/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 52/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 53/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 54/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 55/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 56/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 57/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 58/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 59/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 60/122 2023-12-03T11:23:12.931 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 61/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 62/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 63/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 64/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 65/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 66/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 67/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 68/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 69/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 70/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 71/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 72/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-ply-3.11-14.el9.noarch 73/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 74/122 2023-12-03T11:23:12.932 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 75/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 76/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 77/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 78/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 79/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 80/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 81/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 82/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 83/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 84/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 85/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 86/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 87/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 88/122 2023-12-03T11:23:12.933 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 89/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 90/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 91/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 92/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 93/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 94/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 95/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 96/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 97/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 98/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 99/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 100/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 101/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 102/122 2023-12-03T11:23:12.934 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 103/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 104/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 105/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 106/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 107/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 108/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 109/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 110/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 111/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 112/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 113/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 114/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 115/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : re2-1:20211101-3.el9.x86_64 116/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : s3cmd-2.3.0-4.el9.noarch 117/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 118/122 2023-12-03T11:23:12.935 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 119/122 2023-12-03T11:23:12.936 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 120/122 2023-12-03T11:23:12.936 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 121/122 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 122/122 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout:Upgraded: 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: librados2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: librbd1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.133 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-grafana-dashboards-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-immutable-object-cache-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mds-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-modules-core-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mon-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-osd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-prometheus-alerts-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-radosgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.134 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-selinux-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-test-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-volume-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: cephadm-2:18.0.0-7537.gbcbf278c.el9.noarch 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: jq-1.6-16.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.135 INFO:teuthology.orchestra.run.smithi012.stdout: libcephsqlite-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: librados-devel-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: librgw2-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: lua-5.4.4-4.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-03T11:23:15.136 INFO:teuthology.orchestra.run.smithi012.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-common-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cephfs-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-03T11:23:15.137 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-03T11:23:15.138 INFO:teuthology.orchestra.run.smithi012.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ply-3.11-14.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-03T11:23:15.139 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rados-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rgw-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-03T11:23:15.140 INFO:teuthology.orchestra.run.smithi012.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-fuse-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-mirror-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: rbd-nbd-2:18.0.0-7537.gbcbf278c.el9.x86_64 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: re2-1:20211101-3.el9.x86_64 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: s3cmd-2.3.0-4.el9.noarch 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-03T11:23:15.141 INFO:teuthology.orchestra.run.smithi012.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-03T11:23:15.142 INFO:teuthology.orchestra.run.smithi012.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-03T11:23:15.142 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:15.142 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-03T11:23:15.512 DEBUG:teuthology.parallel:result is None 2023-12-03T11:23:15.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:23:15.665 DEBUG:teuthology.orchestra.run.smithi012:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-03T11:23:15.688 INFO:teuthology.orchestra.run.smithi012.stdout:18.0.0-7537.gbcbf278c.el9 2023-12-03T11:23:15.689 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7537.gbcbf278c.el9 2023-12-03T11:23:15.689 INFO:teuthology.task.install:The correct ceph version 18.0.0-7537.gbcbf278c is installed. 2023-12-03T11:23:15.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:23:15.830 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-03T11:23:15.856 INFO:teuthology.orchestra.run.smithi191.stdout:18.0.0-7537.gbcbf278c.el9 2023-12-03T11:23:15.856 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7537.gbcbf278c.el9 2023-12-03T11:23:15.856 INFO:teuthology.task.install:The correct ceph version 18.0.0-7537.gbcbf278c is installed. 2023-12-03T11:23:15.857 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-03T11:23:15.857 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:15.857 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-03T11:23:15.892 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:15.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-03T11:23:15.930 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-03T11:23:15.930 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:15.930 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2023-12-03T11:23:15.964 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-03T11:23:16.035 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:16.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-12-03T11:23:16.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-03T11:23:16.135 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-03T11:23:16.136 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:16.136 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-03T11:23:16.167 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-03T11:23:16.238 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:16.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-03T11:23:16.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-03T11:23:16.335 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-03T11:23:16.336 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:16.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/stdin-killer 2023-12-03T11:23:16.367 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-03T11:23:16.438 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:16.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2023-12-03T11:23:16.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-03T11:23:16.537 INFO:teuthology.run_tasks:Running task ceph... 2023-12-03T11:23:16.601 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-03T11:23:16.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2023-12-03T11:23:16.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2023-12-03T11:23:16.631 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-03T11:23:16.632 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-03T11:23:16.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-03T11:23:16.702 INFO:tasks.ceph:Creating extra log directories... 2023-12-03T11:23:16.702 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-03T11:23:16.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-03T11:23:16.784 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-03T11:23:16.784 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-03T11:23:16.784 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128', 'branch': 'wip-alimasa-multi-pubsub-etag', 'description': 'rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7476128', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-alimasa-multi-pubsub-etag'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'lockdep': True}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'not have an application enabled', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, 'rgw': {'compression type': 'random', 'ec-data-pool': True, 'erasure_code_profile': {'crush-failure-domain': 'osd', 'k': 3, 'm': 1, 'name': 'testprofile'}, 'frontend': 'beast', 'inline data': False, 'storage classes': 'LUKEWARM, FROZEN'}, 's3tests': {'force-branch': 'ceph-master', 'slow_backend': True, 'storage classes': 'LUKEWARM, FROZEN'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-alimasa-multi-pubsub-etag', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, 'owner': 'scheduled_alimasa@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-alimasa-multi-pubsub-etag', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319', 'targets': {'smithi012.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOYDMCHNBhjGlWzbayJybiaYC93qeZv13jHyTh7c0lvz5KCXrt5k7tAECc6ZHM9rI8bMxWCMauhZfsEvkVYHPEU=', 'smithi191.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZ+1QdwWJLvtMSWw+KNzg1rIK0cQ9aN/t7GZd/tSC1IjlZ68HUn7gFiinnZ4aJuc/FBuzhStjVN2OcEXlQtDeQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['s3cmd'], 'rpm': ['s3cmd']}, 'flavor': 'default', 'sha1': 'bcbf278cf395873c2a482f5a7b2345faf2160319'}}, {'ceph': None}, {'openssl_keys': None}, {'rgw': {'client.0': None}}, {'tox': ['client.0']}, {'workunit': {'clients': {'client.0': ['rgw/run-bucket-check.sh']}}}, {'workunit': {'clients': {'client.0': ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh']}}}, {'workunit': {'clients': {'client.0': ['rgw/test_rgw_s3_mp_reupload.sh']}}}, {'ragweed': {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-reshard.sh']}}}, {'s3tests-java': {'client.0': {'force-branch': 'ceph-master', 'force-repo': 'https://github.com/ceph/java_s3tests.git'}}}, {'s3tests': {'client.0': {'rgw_server': 'client.0'}}}, {'workunit': {'clients': {'client.0': ['rgw/run-versioning.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-03_11:04:43', 'tube': 'smithi', 'user': 'alimasa', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-03T11:23:16.785 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-03T11:23:16.815 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-03T11:23:16.829 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2023-12-03T11:23:16.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2023-12-03T11:23:16.898 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:16.898 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2023-12-03T11:23:16.953 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-03T11:23:16.953 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 2023-12-03T11:23:17.008 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-03T11:23:17.008 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-03T11:23:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.008 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-03 11:23:09.153405666 +0000 2023-12-03T11:23:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-03 11:21:09.156407007 +0000 2023-12-03T11:23:17.009 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-03 11:21:09.156407007 +0000 2023-12-03T11:23:17.009 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-03 11:21:09.156407007 +0000 2023-12-03T11:23:17.009 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-03T11:23:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-03T11:23:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-03T11:23:17.077 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000208709 s, 2.5 MB/s 2023-12-03T11:23:17.078 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-03T11:23:17.134 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-03 11:23:09.153405666 +0000 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-03 11:21:09.676403357 +0000 2023-12-03T11:23:17.189 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-03 11:21:09.676403357 +0000 2023-12-03T11:23:17.190 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-03 11:21:09.676403357 +0000 2023-12-03T11:23:17.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-03T11:23:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-03T11:23:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-03T11:23:17.258 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000187198 s, 2.7 MB/s 2023-12-03T11:23:17.259 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-03T11:23:17.315 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 2023-12-03T11:23:17.370 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-03T11:23:17.370 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.370 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-03T11:23:17.370 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.371 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.371 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-03 11:23:09.154405659 +0000 2023-12-03T11:23:17.371 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-03 11:21:10.193399728 +0000 2023-12-03T11:23:17.371 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-03 11:21:10.193399728 +0000 2023-12-03T11:23:17.371 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-03 11:21:10.193399728 +0000 2023-12-03T11:23:17.371 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-03T11:23:17.439 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-03T11:23:17.439 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-03T11:23:17.439 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000239823 s, 2.1 MB/s 2023-12-03T11:23:17.440 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-03T11:23:17.496 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 2023-12-03T11:23:17.551 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-03T11:23:17.551 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.551 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-03T11:23:17.551 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.551 INFO:teuthology.orchestra.run.smithi012.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.552 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2023-12-03 11:23:09.154405659 +0000 2023-12-03T11:23:17.552 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2023-12-03 11:21:10.708396114 +0000 2023-12-03T11:23:17.552 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2023-12-03 11:21:10.708396114 +0000 2023-12-03T11:23:17.552 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: 2023-12-03 11:21:10.708396114 +0000 2023-12-03T11:23:17.552 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-03T11:23:17.620 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2023-12-03T11:23:17.620 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2023-12-03T11:23:17.620 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000218578 s, 2.3 MB/s 2023-12-03T11:23:17.621 DEBUG:teuthology.orchestra.run.smithi012:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-03T11:23:17.677 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:17.677 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:17.677 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-12-03T11:23:17.694 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-03T11:23:17.694 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-03 11:22:55.847911950 +0000 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-03 11:21:00.430567974 +0000 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-03 11:21:00.430567974 +0000 2023-12-03T11:23:17.750 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-12-03 11:21:00.430567974 +0000 2023-12-03T11:23:17.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-03T11:23:17.817 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-03T11:23:17.817 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-03T11:23:17.817 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000185178 s, 2.8 MB/s 2023-12-03T11:23:17.818 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-03T11:23:17.877 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 683 Links: 1 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-03 11:22:55.847911950 +0000 2023-12-03T11:23:17.930 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-03 11:21:00.959558298 +0000 2023-12-03T11:23:17.931 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-03 11:21:00.959558298 +0000 2023-12-03T11:23:17.931 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-12-03 11:21:00.959558298 +0000 2023-12-03T11:23:17.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-03T11:23:17.996 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-03T11:23:17.997 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-03T11:23:17.997 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000213464 s, 2.4 MB/s 2023-12-03T11:23:17.998 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-03T11:23:18.054 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-03 11:22:55.848911932 +0000 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-03 11:21:01.468548987 +0000 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-03 11:21:01.468548987 +0000 2023-12-03T11:23:18.109 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-12-03 11:21:01.468548987 +0000 2023-12-03T11:23:18.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-03T11:23:18.174 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-03T11:23:18.174 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-03T11:23:18.175 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000207983 s, 2.5 MB/s 2023-12-03T11:23:18.176 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-03T11:23:18.232 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-12-03T11:23:18.289 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-03T11:23:18.289 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 721 Links: 1 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-12-03 11:22:55.848911932 +0000 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-12-03 11:21:01.974539732 +0000 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-12-03 11:21:01.974539732 +0000 2023-12-03T11:23:18.290 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2023-12-03 11:21:01.974539732 +0000 2023-12-03T11:23:18.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-03T11:23:18.355 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-12-03T11:23:18.355 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-12-03T11:23:18.355 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00019401 s, 2.6 MB/s 2023-12-03T11:23:18.356 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-03T11:23:18.412 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:18.413 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-03T11:23:18.413 INFO:tasks.ceph:Generating config... 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] debug rgw = 20 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw bucket counters cache = True 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw torrent flag = True 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] rgw user counters cache = True 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] setgroup = ceph 2023-12-03T11:23:18.414 INFO:tasks.ceph:[client] setuser = ceph 2023-12-03T11:23:18.414 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-03T11:23:18.415 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-03T11:23:18.415 INFO:tasks.ceph:[global] osd_max_pg_log_entries = 10 2023-12-03T11:23:18.415 INFO:tasks.ceph:[global] osd_min_pg_log_entries = 10 2023-12-03T11:23:18.415 INFO:tasks.ceph:[global] rgw override bucket index max shards = 1 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-03T11:23:18.415 INFO:tasks.ceph:[mon] lockdep = True 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-03T11:23:18.415 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] lockdep = True 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-03T11:23:18.416 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-03T11:23:18.416 INFO:tasks.ceph:Setting up mon.a... 2023-12-03T11:23:18.416 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-03T11:23:18.490 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.keyring 2023-12-03T11:23:18.494 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-03T11:23:18.545 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:23:18.576 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.12'), ('mon.c', '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]'), ('mon.b', '172.21.15.191')] 2023-12-03T11:23:18.577 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.191', 'mon client directed command retry': 5, 'ms inject socket failures': 5000, 'osd_max_pg_log_entries': 10, 'osd_min_pg_log_entries': 10, 'rgw override bucket index max shards': 1}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'lockdep': True, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'lockdep': True}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw bucket counters cache': True, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo=', 'rgw torrent flag': True, 'rgw user counters cache': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-03T11:23:18.577 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:18.577 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:23:18.632 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.12 --addv c '[v2:172.21.15.12:3301,v1:172.21.15.12:6790]' --add b 172.21.15.191 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:18.718 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:23:18.718 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:23:18.718 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:setting min_mon_release = pacific 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 16 (pacific) 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:election_strategy: 1 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:23:18.719 INFO:teuthology.orchestra.run.smithi012.stdout:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:23:18.720 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-03T11:23:18.721 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-03T11:23:18.735 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a018b7e2-c099-40b1-a133-6e1bf20b2274... 2023-12-03T11:23:18.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-03T11:23:18.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.837 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.838 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: mon_allow_pool_size_one = true 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-03T11:23:18.839 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.191" 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: mon client directed command retry = 5 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: ms inject socket failures = 5000 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_pg_log_entries = 10 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: osd_min_pg_log_entries = 10 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: rgw override bucket index max shards = 1 2023-12-03T11:23:18.840 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd mclock profile = high_recovery_ops 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.841 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug trim objects = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.842 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bdev async discard = True 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bdev enable discard = True 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore allocator = bitmap 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore block size = 96636764160 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: bluestore fsck on mount = True 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluefs = 1/20 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: debug bluestore = 1/20 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 20 2023-12-03T11:23:18.843 INFO:teuthology.orchestra.run.smithi012.stdout: debug rocksdb = 4/10 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: lockdep = True 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd backfillfull_ratio = 0.85 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd full ratio = 0.9 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd nearfull ratio = 0.8 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: osd failsafe full ratio = 0.95 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: osd objectstore = bluestore 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: osd shutdown pgref assert = True 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:23:18.844 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-03T11:23:18.845 INFO:teuthology.orchestra.run.smithi012.stdout: auth mon ticket ttl = 660 # 11m 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: auth service ticket ttl = 240 # 4m 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: # don't complain about insecure global_id in the test suite 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: # 1m isn't quite enough 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: mon_down_mkfs_grace = 2m 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_filestore_osds = false 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: lockdep = True 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2023-12-03T11:23:18.846 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw = 20 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw bucket counters cache = True 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt require ssl = False 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms backend = testing 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw torrent flag = True 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: rgw user counters cache = True 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: setgroup = ceph 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout: setuser = ceph 2023-12-03T11:23:18.847 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2023-12-03T11:23:18.848 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.c] 2023-12-03T11:23:18.848 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.b] 2023-12-03T11:23:18.850 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2023-12-03T11:23:18.850 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2023-12-03T11:23:18.850 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-03T11:23:18.850 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2023-12-03T11:23:18.850 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.851 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-03T11:23:18.852 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.12,[v2:172.21.15.12:3301,v1:172.21.15.12:6790],172.21.15.191" 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: mon client directed command retry = 5 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: ms inject socket failures = 5000 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: osd_max_pg_log_entries = 10 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: osd_min_pg_log_entries = 10 2023-12-03T11:23:18.853 INFO:teuthology.orchestra.run.smithi191.stdout: rgw override bucket index max shards = 1 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd mclock profile = high_recovery_ops 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2023-12-03T11:23:18.854 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug trim objects = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.855 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: bdev async discard = True 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: bdev enable discard = True 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore allocator = bitmap 2023-12-03T11:23:18.856 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 1/20 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 1/20 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 4/10 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: lockdep = True 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2023-12-03T11:23:18.857 INFO:teuthology.orchestra.run.smithi191.stdout: osd shutdown pgref assert = True 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-12-03T11:23:18.858 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-03T11:23:18.859 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: lockdep = True 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: debug rgw = 20 2023-12-03T11:23:18.860 INFO:teuthology.orchestra.run.smithi191.stdout: rgw bucket counters cache = True 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: rgw crypt require ssl = False 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: rgw crypt s3 kms backend = testing 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= testkey-2=aWIKTWFrZWZpbGUKbWFuCm91dApzcmMKVGVzdGluZwo= 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: rgw torrent flag = True 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: rgw user counters cache = True 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: setgroup = ceph 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout: setuser = ceph 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2023-12-03T11:23:18.861 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2023-12-03T11:23:18.864 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-03T11:23:18.864 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-03T11:23:18.964 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-03T11:23:18.964 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:18.964 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-03T11:23:18.979 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:18.979 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-03T11:23:19.034 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:23:19.034 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:19.034 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-03T11:23:19.034 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:23:19.120 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:19.120 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:19.175 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:23:19.175 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:19.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-03T11:23:19.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-03T11:23:19.217 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:19.218 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:19.273 INFO:tasks.ceph:Setting up mon nodes... 2023-12-03T11:23:19.274 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-03T11:23:19.274 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-03T11:23:19.335 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-03T11:23:19.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:23:19.398 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-03T11:23:19.400 INFO:tasks.ceph:Setting up mds nodes... 2023-12-03T11:23:19.400 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-03T11:23:19.401 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-03T11:23:19.447 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-03T11:23:19.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-03T11:23:19.509 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-03T11:23:19.524 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-03T11:23:19.524 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-03T11:23:19.524 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-03T11:23:19.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:19.555 INFO:tasks.ceph:role: osd.0 2023-12-03T11:23:19.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:23:19.555 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:19.633 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:19.634 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:20.133 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-03T11:23:20.135 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-03T11:23:20.135 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-03T11:23:20.265 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-03T11:23:20.304 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-03T11:23:20.375 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:20.376 INFO:tasks.ceph:role: osd.1 2023-12-03T11:23:20.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:23:20.376 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-03T11:23:20.454 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:20.454 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:20.455 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:20.905 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-03T11:23:20.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-03T11:23:20.907 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-03T11:23:21.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-03T11:23:21.036 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-03T11:23:21.107 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:21.107 INFO:tasks.ceph:role: osd.2 2023-12-03T11:23:21.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:23:21.107 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-03T11:23:21.185 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:21.185 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:21.186 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:21.691 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-03T11:23:21.693 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-03T11:23:21.694 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-03T11:23:21.836 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-03T11:23:21.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-03T11:23:21.945 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:21.945 INFO:tasks.ceph:role: osd.3 2023-12-03T11:23:21.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:23:21.945 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:22.024 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:22.025 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:22.025 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:22.540 INFO:teuthology.orchestra.run.smithi012.stdout:Discarding blocks...Done. 2023-12-03T11:23:22.542 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2023-12-03T11:23:22.542 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-03T11:23:22.641 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-03T11:23:22.680 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.800+0000 7fd135f39500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-03T11:23:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.801+0000 7fd135f39500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-03T11:23:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.801+0000 7fd135f39500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:23:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.801+0000 7fd135f39500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:23:22.802 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.801+0000 7fd135f39500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-03T11:23:22.803 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:22.801+0000 7fd135f39500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-03T11:23:25.656 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-03T11:23:25.692 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.787+0000 7ff966b07500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-03T11:23:25.789 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.788+0000 7ff966b07500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-03T11:23:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.788+0000 7ff966b07500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:23:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.788+0000 7ff966b07500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:23:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.788+0000 7ff966b07500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-03T11:23:25.790 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:25.788+0000 7ff966b07500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-03T11:23:28.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-03T11:23:28.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.777+0000 7ff2c4b1a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.778+0000 7ff2c4b1a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.778+0000 7ff2c4b1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.778+0000 7ff2c4b1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.778+0000 7ff2c4b1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-03T11:23:28.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:28.778+0000 7ff2c4b1a500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-03T11:23:31.647 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-03T11:23:31.683 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.777+0000 7f112a41a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.778+0000 7f112a41a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.778+0000 7f112a41a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.778+0000 7f112a41a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.778+0000 7f112a41a500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-03T11:23:31.779 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:23:31.778+0000 7f112a41a500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-03T11:23:34.634 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-03T11:23:34.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-03T11:23:34.702 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:34.702 INFO:tasks.ceph:role: osd.4 2023-12-03T11:23:34.702 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:23:34.702 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:34.780 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:35.263 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-03T11:23:35.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-03T11:23:35.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-03T11:23:35.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-03T11:23:35.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-03T11:23:35.502 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:35.502 INFO:tasks.ceph:role: osd.5 2023-12-03T11:23:35.502 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:23:35.502 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:35.578 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:35.579 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:35.579 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:35.579 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:35.579 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:36.033 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-03T11:23:36.035 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-03T11:23:36.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-03T11:23:36.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-03T11:23:36.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-03T11:23:36.228 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:36.228 INFO:tasks.ceph:role: osd.6 2023-12-03T11:23:36.228 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:23:36.229 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-03T11:23:36.303 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:36.304 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:36.792 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-03T11:23:36.794 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-03T11:23:36.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-03T11:23:36.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-03T11:23:36.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-03T11:23:37.034 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-03T11:23:37.034 INFO:tasks.ceph:role: osd.7 2023-12-03T11:23:37.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:23:37.034 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-03T11:23:37.110 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-03T11:23:37.111 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-03T11:23:37.111 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-03T11:23:37.613 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-12-03T11:23:37.615 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-12-03T11:23:37.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-03T11:23:37.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-03T11:23:37.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.881+0000 7f69c994a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.882+0000 7f69c994a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.882+0000 7f69c994a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.882+0000 7f69c994a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.882+0000 7f69c994a500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-03T11:23:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:37.882+0000 7f69c994a500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-03T11:23:40.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-03T11:23:40.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.849+0000 7fa9e1590500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.849+0000 7fa9e1590500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.849+0000 7fa9e1590500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.849+0000 7fa9e1590500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.849+0000 7fa9e1590500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-03T11:23:40.851 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:40.850+0000 7fa9e1590500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-03T11:23:43.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-03T11:23:43.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.816+0000 7f87cf81f500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.816+0000 7f87cf81f500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.816+0000 7f87cf81f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.816+0000 7f87cf81f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.816+0000 7f87cf81f500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-03T11:23:43.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:43.817+0000 7f87cf81f500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-03T11:23:46.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-03T11:23:46.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:46.790 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-03T11:23:46.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-03T11:23:46.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:23:46.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:23:46.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-03T11:23:46.791 INFO:teuthology.orchestra.run.smithi191.stderr:2023-12-03T11:23:46.788+0000 7f2540e14500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-03T11:23:49.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-03T11:23:49.690 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-03T11:23:49.690 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:49.690 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-03T11:23:49.726 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:49.726 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-03T11:23:49.799 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:49.799 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-03T11:23:49.873 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:49.873 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-03T11:23:49.945 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:49.946 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-03T11:23:50.018 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-03T11:23:50.050 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-03T11:23:50.120 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-03T11:23:50.189 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-03T11:23:50.257 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-03T11:23:50.326 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:50.326 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-03T11:23:50.341 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:50.341 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-03T11:23:50.381 INFO:tasks.ceph:Adding keys to all mons... 2023-12-03T11:23:50.381 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-03T11:23:50.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-03T11:23:50.424 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.y] 2023-12-03T11:23:50.424 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAnZWxllX8BFBAA2487hmh4lUCE7CuY1iOo/A== 2023-12-03T11:23:50.424 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2023-12-03T11:23:50.424 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAqZWxlAI7JLxAAh/3i/QT/eGaDbZtI6uj8pA== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAtZWxlz50ELxAA7ZfyHuwhVmRWy4RtG1L0tQ== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAwZWxlAQ9rLhAAmwnT/EImQ+Z2iGVmjLCaPA== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.3] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAzZWxlaQZrLhAAhSRCprGCX02Ql0yn4obxxg== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAnZWxl3fW3FxAAL5bwudSsvvd5MQJ0xn6QwA== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.4] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA5ZWxlXjWdNBAARVxArs/AlsOY8N+ly2b6nA== 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.5] 2023-12-03T11:23:50.425 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA8ZWxlxi6vMhAA3voPzD0ADcqyEQWoeAJRlQ== 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.6] 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA/ZWxlXW26MBAABo4tQlt5OZ/15Qoly26Rig== 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.7] 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBCZWxlxBwXLxAAZ6qY+fJNfsgYrplFuk1soA== 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAnZWxlE0OsGhAAw0hkFa9gNz7vETNbb+VaRg== 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2023-12-03T11:23:50.426 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQAnZWxlRCtTHhAA9XVP16vyRBaiN2yk0FXzdQ== 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAnZWxllX8BFBAA2487hmh4lUCE7CuY1iOo/A== 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAqZWxlAI7JLxAAh/3i/QT/eGaDbZtI6uj8pA== 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2023-12-03T11:23:50.449 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAtZWxlz50ELxAA7ZfyHuwhVmRWy4RtG1L0tQ== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAwZWxlAQ9rLhAAmwnT/EImQ+Z2iGVmjLCaPA== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAzZWxlaQZrLhAAhSRCprGCX02Ql0yn4obxxg== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAnZWxl3fW3FxAAL5bwudSsvvd5MQJ0xn6QwA== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQA5ZWxlXjWdNBAARVxArs/AlsOY8N+ly2b6nA== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQA8ZWxlxi6vMhAA3voPzD0ADcqyEQWoeAJRlQ== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQA/ZWxlXW26MBAABo4tQlt5OZ/15Qoly26Rig== 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2023-12-03T11:23:50.450 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQBCZWxlxBwXLxAAZ6qY+fJNfsgYrplFuk1soA== 2023-12-03T11:23:50.451 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2023-12-03T11:23:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAnZWxlE0OsGhAAw0hkFa9gNz7vETNbb+VaRg== 2023-12-03T11:23:50.451 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2023-12-03T11:23:50.451 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAnZWxlRCtTHhAA9XVP16vyRBaiN2yk0FXzdQ== 2023-12-03T11:23:50.452 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:23:50.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:23:50.544 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.601 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.657 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.713 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:23:50.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-03T11:23:50.824 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.879 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.935 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.991 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:50.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-03T11:23:51.047 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:23:51.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:23:51.106 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:23:51.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-03T11:23:51.164 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-03T11:23:51.164 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-03T11:23:51.194 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:23:51.296 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-03T11:23:51.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-03T11:23:51.397 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:23:51.498 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-03T11:23:51.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-03T11:23:51.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-03T11:23:51.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-03T11:23:51.678 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:51.681 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-03T11:23:51.733 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-03T11:23:51.734 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-03T11:23:51.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-03T11:23:51.736 INFO:tasks.ceph.mon.a:Started 2023-12-03T11:23:51.736 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-03T11:23:51.736 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-03T11:23:51.738 INFO:tasks.ceph.mon.c:Started 2023-12-03T11:23:51.738 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-03T11:23:51.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-03T11:23:51.776 INFO:tasks.ceph.mon.b:Started 2023-12-03T11:23:51.776 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-03T11:23:51.777 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-03T11:23:51.777 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-03T11:23:51.779 INFO:tasks.ceph.mgr.y:Started 2023-12-03T11:23:51.779 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-03T11:23:51.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-03T11:23:51.781 INFO:tasks.ceph.mgr.x:Started 2023-12-03T11:23:51.781 INFO:tasks.ceph:Setting crush tunables to default 2023-12-03T11:23:51.781 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd crush tunables default 2023-12-03T11:23:57.189 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2023-12-03T11:23:57.202 INFO:tasks.ceph:check_enable_crimson: False 2023-12-03T11:23:57.202 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-03T11:23:57.205 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:57.205 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-03T11:23:57.237 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:57.237 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-03T11:23:57.308 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:57.308 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-03T11:23:57.379 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:23:57.379 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-03T11:23:57.450 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:57.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-03T11:23:57.477 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:57.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-03T11:23:57.545 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:57.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-03T11:23:57.575 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:57.574+0000 7fb5e102b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-03T11:23:57.601 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-12-03T11:23:57.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-03T11:23:57.628 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:57.627+0000 7fb5e102b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-03T11:23:57.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new b0833b12-5074-4524-8e87-e10fdb972eb0 0 2023-12-03T11:23:57.880 INFO:teuthology.orchestra.run.smithi191.stdout:0 2023-12-03T11:23:57.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 864651e1-892e-40f9-9dc0-b7b3aac10e6e 1 2023-12-03T11:23:58.080 INFO:teuthology.orchestra.run.smithi191.stdout:1 2023-12-03T11:23:58.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new e7379202-7fe3-4e18-9f99-da8edfc6734c 2 2023-12-03T11:23:58.179 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.178+0000 7fb5e102b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-03T11:23:58.236 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.235+0000 7fb5e102b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-03T11:23:58.281 INFO:teuthology.orchestra.run.smithi191.stdout:2 2023-12-03T11:23:58.282 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.281+0000 7fb5e102b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-03T11:23:58.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new a439ad5e-6bf0-431d-96c2-6cf81d89e967 3 2023-12-03T11:23:58.340 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.339+0000 7fb5e102b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-03T11:23:58.343 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:58.342+0000 7f267462b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-03T11:23:58.382 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.381+0000 7fb5e102b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-03T11:23:58.396 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:58.395+0000 7f267462b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-03T11:23:58.481 INFO:teuthology.orchestra.run.smithi191.stdout:3 2023-12-03T11:23:58.490 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.489+0000 7fb5e102b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-03T11:23:58.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 4c91afb1-4729-450f-b8f1-6c355b52603a 4 2023-12-03T11:23:58.564 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.563+0000 7fb5e102b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-03T11:23:58.612 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.611+0000 7fb5e102b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-03T11:23:58.665 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.664+0000 7fb5e102b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-03T11:23:58.677 INFO:teuthology.orchestra.run.smithi191.stdout:4 2023-12-03T11:23:58.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new e48238ad-f8fb-4664-a757-03d6eec75f41 5 2023-12-03T11:23:58.717 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:58.716+0000 7fb5e102b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-03T11:23:58.874 INFO:teuthology.orchestra.run.smithi191.stdout:5 2023-12-03T11:23:58.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new 2a212dc9-0b4a-4f23-97f2-1a11e1c551bb 6 2023-12-03T11:23:58.953 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:58.952+0000 7f267462b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-03T11:23:59.008 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.006+0000 7f267462b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-03T11:23:59.054 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.053+0000 7f267462b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-03T11:23:59.110 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.109+0000 7f267462b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-03T11:23:59.115 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.114+0000 7fb5e102b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-03T11:23:59.116 INFO:teuthology.orchestra.run.smithi191.stdout:6 2023-12-03T11:23:59.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd new acbf307b-a989-47ed-af4f-ffd17549ee69 7 2023-12-03T11:23:59.155 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.154+0000 7f267462b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-03T11:23:59.260 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.259+0000 7f267462b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-03T11:23:59.266 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.265+0000 7fb5e102b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-03T11:23:59.322 INFO:teuthology.orchestra.run.smithi191.stdout:7 2023-12-03T11:23:59.333 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-03T11:23:59.333 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-03T11:23:59.334 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.333+0000 7f267462b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-03T11:23:59.336 INFO:tasks.ceph.osd.0:Started 2023-12-03T11:23:59.336 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-03T11:23:59.336 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-03T11:23:59.338 INFO:tasks.ceph.osd.1:Started 2023-12-03T11:23:59.338 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-03T11:23:59.338 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-03T11:23:59.342 INFO:tasks.ceph.osd.2:Started 2023-12-03T11:23:59.342 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-03T11:23:59.342 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-03T11:23:59.345 INFO:tasks.ceph.osd.3:Started 2023-12-03T11:23:59.345 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-03T11:23:59.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-03T11:23:59.347 INFO:tasks.ceph.osd.4:Started 2023-12-03T11:23:59.347 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-03T11:23:59.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-03T11:23:59.349 INFO:tasks.ceph.osd.5:Started 2023-12-03T11:23:59.349 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-03T11:23:59.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-03T11:23:59.351 INFO:tasks.ceph.osd.6:Started 2023-12-03T11:23:59.351 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-03T11:23:59.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-03T11:23:59.354 INFO:tasks.ceph.osd.7:Started 2023-12-03T11:23:59.354 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:23:59.407 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.406+0000 7fb5e102b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-03T11:23:59.408 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.407+0000 7f267462b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-03T11:23:59.463 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.461+0000 7fb5e102b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-03T11:23:59.484 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.482+0000 7f267462b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-03T11:23:59.534 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.533+0000 7f267462b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-03T11:23:59.559 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.558+0000 7fb5e102b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:23:59.560 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:23:59.561 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":10,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:23:59.319754+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:23:59.572 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-03T11:23:59.572 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-03T11:23:59.653 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.652+0000 7fb5e102b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-03T11:23:59.792 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.791+0000 7fb5e102b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:23:59.834 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.833+0000 7fb5e102b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-03T11:23:59.932 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:23:59.931+0000 7f267462b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-03T11:23:59.950 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:23:59.949+0000 7f47d5d1c500 -1 Falling back to public interface 2023-12-03T11:23:59.951 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:23:59.949+0000 7f8d69c12500 -1 Falling back to public interface 2023-12-03T11:23:59.951 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:23:59.950+0000 7f790a861500 -1 Falling back to public interface 2023-12-03T11:23:59.955 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:23:59.954+0000 7fccab382500 -1 Falling back to public interface 2023-12-03T11:23:59.958 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:23:59.957+0000 7fa50280f500 -1 Falling back to public interface 2023-12-03T11:23:59.967 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:23:59.966+0000 7fc8ff010500 -1 Falling back to public interface 2023-12-03T11:23:59.969 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:23:59.968+0000 7f5d78611500 -1 Falling back to public interface 2023-12-03T11:23:59.979 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:23:59.978+0000 7fb5e102b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-03T11:23:59.988 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:23:59.987+0000 7f930f4da500 -1 Falling back to public interface 2023-12-03T11:24:00.076 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.075+0000 7f267462b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-03T11:24:00.197 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.196+0000 7f267462b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-03T11:24:00.244 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.243+0000 7f267462b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-03T11:24:00.346 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.345+0000 7f267462b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:24:00.378 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:24:00.446 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.445+0000 7f267462b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-03T11:24:00.542 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-03T11:24:00.542 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":10,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:23:59.319754+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:24:00.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-03T11:24:00.595 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.594+0000 7f267462b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-03T11:24:00.639 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.638+0000 7f267462b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-03T11:24:00.787 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:00.786+0000 7f267462b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-03T11:24:00.807 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:24:00.806+0000 7fb5e102b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-03T11:24:01.073 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:24:01.072+0000 7fb5e102b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-03T11:24:01.124 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:24:01.123+0000 7fb5e102b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-03T11:24:01.222 INFO:tasks.ceph.mgr.x.smithi191.stderr:2023-12-03T11:24:01.221+0000 7fb5e102b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-03T11:24:01.612 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:01.611+0000 7f267462b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-03T11:24:01.896 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:01.895+0000 7f267462b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-03T11:24:01.950 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:01.948+0000 7f267462b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-03T11:24:02.054 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:02.053+0000 7f267462b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-03T11:24:03.078 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:03.077+0000 7f26152d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-03T11:24:03.078 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:24:03.077+0000 7f26152d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-03T11:24:03.408 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:24:03.407+0000 7f790a861500 -1 osd.4 0 log_to_monitors true 2023-12-03T11:24:03.414 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:24:03.412+0000 7f8d69c12500 -1 osd.1 0 log_to_monitors true 2023-12-03T11:24:03.457 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:24:03.455+0000 7fccab382500 -1 osd.2 0 log_to_monitors true 2023-12-03T11:24:03.473 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:24:03.472+0000 7f47d5d1c500 -1 osd.5 0 log_to_monitors true 2023-12-03T11:24:03.481 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:24:03.480+0000 7fc8ff010500 -1 osd.7 0 log_to_monitors true 2023-12-03T11:24:03.483 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:24:03.482+0000 7fa50280f500 -1 osd.6 0 log_to_monitors true 2023-12-03T11:24:03.484 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:24:03.483+0000 7f5d78611500 -1 osd.0 0 log_to_monitors true 2023-12-03T11:24:03.497 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:24:03.496+0000 7f930f4da500 -1 osd.3 0 log_to_monitors true 2023-12-03T11:24:07.064 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:24:07.062+0000 7fcca6aef640 -1 osd.2 0 waiting for initial osdmap 2023-12-03T11:24:07.077 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:24:07.076+0000 7fcc9f127640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.152 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:24:07.151+0000 7f7905fcc640 -1 osd.4 0 waiting for initial osdmap 2023-12-03T11:24:07.166 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:24:07.165+0000 7f78fe604640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.172 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:24:07.171+0000 7f47d1487640 -1 osd.5 0 waiting for initial osdmap 2023-12-03T11:24:07.180 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:24:07.179+0000 7f47c9abf640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.183 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:24:07.182+0000 7fc8fa26c640 -1 osd.7 0 waiting for initial osdmap 2023-12-03T11:24:07.190 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:24:07.189+0000 7f8d66684640 -1 osd.1 0 waiting for initial osdmap 2023-12-03T11:24:07.191 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:24:07.190+0000 7fc8f28a4640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.194 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:24:07.193+0000 7fa4fd8ab640 -1 osd.6 0 waiting for initial osdmap 2023-12-03T11:24:07.200 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:24:07.199+0000 7f8d5d484640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.208 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:24:07.207+0000 7fa4f5ee3640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.240 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:24:07.239+0000 7f5d74ff3640 -1 osd.0 0 waiting for initial osdmap 2023-12-03T11:24:07.250 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:24:07.249+0000 7f5d6bdf3640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.260 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:24:07.259+0000 7f930ac47640 -1 osd.3 0 waiting for initial osdmap 2023-12-03T11:24:07.272 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:24:07.271+0000 7f930327f640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-03T11:24:07.358 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:24:07.639 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-03T11:24:07.640 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":12,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:24:05.079668+0000","last_up_change":"0.000000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:24:07.651 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-03T11:24:14.457 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-03T11:24:14.749 INFO:teuthology.misc.health.smithi012.stdout: 2023-12-03T11:24:14.749 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":16,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:24:11.071738+0000","last_up_change":"2023-12-03T11:24:08.066796+0000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:24:08.129946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6817","nonce":1420251316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6819","nonce":1420251316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6823","nonce":1420251316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6821","nonce":1420251316}]},"public_addr":"172.21.15.12:6817/1420251316","cluster_addr":"172.21.15.12:6819/1420251316","heartbeat_back_addr":"172.21.15.12:6823/1420251316","heartbeat_front_addr":"172.21.15.12:6821/1420251316","state":["exists","up"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6801","nonce":2316181860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6803","nonce":2316181860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6807","nonce":2316181860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6805","nonce":2316181860}]},"public_addr":"172.21.15.12:6801/2316181860","cluster_addr":"172.21.15.12:6803/2316181860","heartbeat_back_addr":"172.21.15.12:6807/2316181860","heartbeat_front_addr":"172.21.15.12:6805/2316181860","state":["exists","up"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6809","nonce":3792383689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6811","nonce":3792383689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6815","nonce":3792383689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6813","nonce":3792383689}]},"public_addr":"172.21.15.12:6809/3792383689","cluster_addr":"172.21.15.12:6811/3792383689","heartbeat_back_addr":"172.21.15.12:6815/3792383689","heartbeat_front_addr":"172.21.15.12:6813/3792383689","state":["exists","up"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6825","nonce":518024637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6827","nonce":518024637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6831","nonce":518024637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6829","nonce":518024637}]},"public_addr":"172.21.15.12:6825/518024637","cluster_addr":"172.21.15.12:6827/518024637","heartbeat_back_addr":"172.21.15.12:6831/518024637","heartbeat_front_addr":"172.21.15.12:6829/518024637","state":["exists","up"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6805","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6807","nonce":3310030881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6811","nonce":3310030881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6815","nonce":3310030881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6813","nonce":3310030881}]},"public_addr":"172.21.15.191:6807/3310030881","cluster_addr":"172.21.15.191:6811/3310030881","heartbeat_back_addr":"172.21.15.191:6815/3310030881","heartbeat_front_addr":"172.21.15.191:6813/3310030881","state":["exists","up"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6801","nonce":3759890485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6803","nonce":3759890485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6810","nonce":3759890485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6806","nonce":3759890485}]},"public_addr":"172.21.15.191:6801/3759890485","cluster_addr":"172.21.15.191:6803/3759890485","heartbeat_back_addr":"172.21.15.191:6810/3759890485","heartbeat_front_addr":"172.21.15.191:6806/3759890485","state":["exists","up"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6817","nonce":2577115973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6819","nonce":2577115973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6823","nonce":2577115973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6821","nonce":2577115973}]},"public_addr":"172.21.15.191:6817/2577115973","cluster_addr":"172.21.15.191:6819/2577115973","heartbeat_back_addr":"172.21.15.191:6823/2577115973","heartbeat_front_addr":"172.21.15.191:6821/2577115973","state":["exists","up"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6825","nonce":2936799433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6827","nonce":2936799433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6831","nonce":2936799433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6829","nonce":2936799433}]},"public_addr":"172.21.15.191:6825/2936799433","cluster_addr":"172.21.15.191:6827/2936799433","heartbeat_back_addr":"172.21.15.191:6831/2936799433","heartbeat_front_addr":"172.21.15.191:6829/2936799433","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.513466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.393906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.496239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.449624+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.376875+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.502149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.511904+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.478095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:24:14.760 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-03T11:24:14.760 INFO:tasks.ceph:Creating RBD pool 2023-12-03T11:24:14.760 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-03T11:24:15.109 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2023-12-03T11:24:15.121 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-03T11:24:16.112 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-03T11:24:16.128 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-03T11:24:16.129 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-03T11:24:16.129 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-03T11:24:16.129 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-03T11:24:16.130 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-03T11:24:16.490 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:16.501 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":6,"active_gid":4122,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6832","nonce":3565462874},{"type":"v1","addr":"172.21.15.12:6833","nonce":3565462874}]},"active_addr":"172.21.15.12:6833/3565462874","active_change":"2023-12-03T11:24:02.067730+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2823872564}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":988485449}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3240430872}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":1453831910}]}]} 2023-12-03T11:24:16.505 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-03T11:24:16.505 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-03T11:24:16.505 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:24:16.799 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:16.799 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:24:16.102383+0000","last_up_change":"2023-12-03T11:24:08.066796+0000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:24:08.129946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:24:15.082484+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6817","nonce":1420251316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6819","nonce":1420251316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6823","nonce":1420251316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6821","nonce":1420251316}]},"public_addr":"172.21.15.12:6817/1420251316","cluster_addr":"172.21.15.12:6819/1420251316","heartbeat_back_addr":"172.21.15.12:6823/1420251316","heartbeat_front_addr":"172.21.15.12:6821/1420251316","state":["exists","up"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6801","nonce":2316181860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6803","nonce":2316181860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6807","nonce":2316181860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6805","nonce":2316181860}]},"public_addr":"172.21.15.12:6801/2316181860","cluster_addr":"172.21.15.12:6803/2316181860","heartbeat_back_addr":"172.21.15.12:6807/2316181860","heartbeat_front_addr":"172.21.15.12:6805/2316181860","state":["exists","up"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6809","nonce":3792383689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6811","nonce":3792383689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6815","nonce":3792383689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6813","nonce":3792383689}]},"public_addr":"172.21.15.12:6809/3792383689","cluster_addr":"172.21.15.12:6811/3792383689","heartbeat_back_addr":"172.21.15.12:6815/3792383689","heartbeat_front_addr":"172.21.15.12:6813/3792383689","state":["exists","up"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6825","nonce":518024637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6827","nonce":518024637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6831","nonce":518024637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6829","nonce":518024637}]},"public_addr":"172.21.15.12:6825/518024637","cluster_addr":"172.21.15.12:6827/518024637","heartbeat_back_addr":"172.21.15.12:6831/518024637","heartbeat_front_addr":"172.21.15.12:6829/518024637","state":["exists","up"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6805","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6807","nonce":3310030881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6811","nonce":3310030881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6815","nonce":3310030881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6813","nonce":3310030881}]},"public_addr":"172.21.15.191:6807/3310030881","cluster_addr":"172.21.15.191:6811/3310030881","heartbeat_back_addr":"172.21.15.191:6815/3310030881","heartbeat_front_addr":"172.21.15.191:6813/3310030881","state":["exists","up"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6801","nonce":3759890485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6803","nonce":3759890485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6810","nonce":3759890485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6806","nonce":3759890485}]},"public_addr":"172.21.15.191:6801/3759890485","cluster_addr":"172.21.15.191:6803/3759890485","heartbeat_back_addr":"172.21.15.191:6810/3759890485","heartbeat_front_addr":"172.21.15.191:6806/3759890485","state":["exists","up"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6817","nonce":2577115973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6819","nonce":2577115973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6823","nonce":2577115973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6821","nonce":2577115973}]},"public_addr":"172.21.15.191:6817/2577115973","cluster_addr":"172.21.15.191:6819/2577115973","heartbeat_back_addr":"172.21.15.191:6823/2577115973","heartbeat_front_addr":"172.21.15.191:6821/2577115973","state":["exists","up"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6825","nonce":2936799433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6827","nonce":2936799433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6831","nonce":2936799433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6829","nonce":2936799433}]},"public_addr":"172.21.15.191:6825/2936799433","cluster_addr":"172.21.15.191:6827/2936799433","heartbeat_back_addr":"172.21.15.191:6831/2936799433","heartbeat_front_addr":"172.21.15.191:6829/2936799433","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.513466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.393906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.496239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.449624+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.376875+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.502149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.511904+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.478095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:24:16.812 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-03T11:24:16.812 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-03T11:24:17.112 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:17.112 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":18,"fsid":"a018b7e2-c099-40b1-a133-6e1bf20b2274","created":"2023-12-03T11:23:57.093537+0000","modified":"2023-12-03T11:24:16.102383+0000","last_up_change":"2023-12-03T11:24:08.066796+0000","last_in_change":"2023-12-03T11:23:59.319754+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-03T11:24:08.129946+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-03T11:24:15.082484+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b0833b12-5074-4524-8e87-e10fdb972eb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6817","nonce":1420251316}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6819","nonce":1420251316}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6823","nonce":1420251316}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":1420251316},{"type":"v1","addr":"172.21.15.12:6821","nonce":1420251316}]},"public_addr":"172.21.15.12:6817/1420251316","cluster_addr":"172.21.15.12:6819/1420251316","heartbeat_back_addr":"172.21.15.12:6823/1420251316","heartbeat_front_addr":"172.21.15.12:6821/1420251316","state":["exists","up"]},{"osd":1,"uuid":"864651e1-892e-40f9-9dc0-b7b3aac10e6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6801","nonce":2316181860}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6803","nonce":2316181860}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6807","nonce":2316181860}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6804","nonce":2316181860},{"type":"v1","addr":"172.21.15.12:6805","nonce":2316181860}]},"public_addr":"172.21.15.12:6801/2316181860","cluster_addr":"172.21.15.12:6803/2316181860","heartbeat_back_addr":"172.21.15.12:6807/2316181860","heartbeat_front_addr":"172.21.15.12:6805/2316181860","state":["exists","up"]},{"osd":2,"uuid":"e7379202-7fe3-4e18-9f99-da8edfc6734c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6808","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6809","nonce":3792383689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6811","nonce":3792383689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6815","nonce":3792383689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6812","nonce":3792383689},{"type":"v1","addr":"172.21.15.12:6813","nonce":3792383689}]},"public_addr":"172.21.15.12:6809/3792383689","cluster_addr":"172.21.15.12:6811/3792383689","heartbeat_back_addr":"172.21.15.12:6815/3792383689","heartbeat_front_addr":"172.21.15.12:6813/3792383689","state":["exists","up"]},{"osd":3,"uuid":"a439ad5e-6bf0-431d-96c2-6cf81d89e967","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6825","nonce":518024637}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6826","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6827","nonce":518024637}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6830","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6831","nonce":518024637}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6828","nonce":518024637},{"type":"v1","addr":"172.21.15.12:6829","nonce":518024637}]},"public_addr":"172.21.15.12:6825/518024637","cluster_addr":"172.21.15.12:6827/518024637","heartbeat_back_addr":"172.21.15.12:6831/518024637","heartbeat_front_addr":"172.21.15.12:6829/518024637","state":["exists","up"]},{"osd":4,"uuid":"4c91afb1-4729-450f-b8f1-6c355b52603a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6805","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6807","nonce":3310030881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6809","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6811","nonce":3310030881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6815","nonce":3310030881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":3310030881},{"type":"v1","addr":"172.21.15.191:6813","nonce":3310030881}]},"public_addr":"172.21.15.191:6807/3310030881","cluster_addr":"172.21.15.191:6811/3310030881","heartbeat_back_addr":"172.21.15.191:6815/3310030881","heartbeat_front_addr":"172.21.15.191:6813/3310030881","state":["exists","up"]},{"osd":5,"uuid":"e48238ad-f8fb-4664-a757-03d6eec75f41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6801","nonce":3759890485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6803","nonce":3759890485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6810","nonce":3759890485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":3759890485},{"type":"v1","addr":"172.21.15.191:6806","nonce":3759890485}]},"public_addr":"172.21.15.191:6801/3759890485","cluster_addr":"172.21.15.191:6803/3759890485","heartbeat_back_addr":"172.21.15.191:6810/3759890485","heartbeat_front_addr":"172.21.15.191:6806/3759890485","state":["exists","up"]},{"osd":6,"uuid":"2a212dc9-0b4a-4f23-97f2-1a11e1c551bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6817","nonce":2577115973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6819","nonce":2577115973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6823","nonce":2577115973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2577115973},{"type":"v1","addr":"172.21.15.191:6821","nonce":2577115973}]},"public_addr":"172.21.15.191:6817/2577115973","cluster_addr":"172.21.15.191:6819/2577115973","heartbeat_back_addr":"172.21.15.191:6823/2577115973","heartbeat_front_addr":"172.21.15.191:6821/2577115973","state":["exists","up"]},{"osd":7,"uuid":"acbf307b-a989-47ed-af4f-ffd17549ee69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6825","nonce":2936799433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6827","nonce":2936799433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6831","nonce":2936799433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":2936799433},{"type":"v1","addr":"172.21.15.191:6829","nonce":2936799433}]},"public_addr":"172.21.15.191:6825/2936799433","cluster_addr":"172.21.15.191:6827/2936799433","heartbeat_back_addr":"172.21.15.191:6831/2936799433","heartbeat_front_addr":"172.21.15.191:6829/2936799433","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.513466+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.393906+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.496239+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.449624+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.376875+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.502149+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.511904+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-03T11:24:04.478095+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-03T11:24:17.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-03T11:24:17.126 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-03T11:24:17.127 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-03T11:24:17.127 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-03T11:24:17.127 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-03T11:24:17.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-03T11:24:17.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-03T11:24:17.128 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-03T11:24:17.388 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.389 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-03T11:24:17.389 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.390 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-03T11:24:17.391 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2023-12-03T11:24:17.391 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:24:17.392 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.392 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-03T11:24:17.393 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2023-12-03T11:24:17.393 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-03T11:24:17.394 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.394 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:24:17.395 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.397 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-03T11:24:17.398 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:17.399 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-03T11:24:17.977 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:17.978 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:17.999 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.4 2023-12-03T11:24:18.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.3 2023-12-03T11:24:18.021 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:18.021 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:18.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2023-12-03T11:24:18.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2023-12-03T11:24:18.053 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2023-12-03T11:24:18.055 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:18.065 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:18.067 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-12-03T11:24:18.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.0 2023-12-03T11:24:18.071 INFO:teuthology.orchestra.run.smithi012.stdout:55834574851 2023-12-03T11:24:18.080 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.2 2023-12-03T11:24:18.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2023-12-03T11:24:19.000 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-03T11:24:19.001 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-03T11:24:19.037 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-03T11:24:19.037 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-03T11:24:19.068 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-03T11:24:19.069 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-03T11:24:19.081 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-03T11:24:19.085 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-03T11:24:19.565 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.579 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.585 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.4 2023-12-03T11:24:19.585 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.599 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.3 2023-12-03T11:24:19.599 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.616 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.629 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2023-12-03T11:24:19.629 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.636 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.645 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.654 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2023-12-03T11:24:19.654 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.658 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2023-12-03T11:24:19.658 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.679 INFO:teuthology.orchestra.run.smithi012.stdout:55834574852 2023-12-03T11:24:19.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.0 2023-12-03T11:24:19.692 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.700 INFO:teuthology.orchestra.run.smithi012.stdout:55834574854 2023-12-03T11:24:19.703 INFO:teuthology.orchestra.run.smithi012.stdout:55834574853 2023-12-03T11:24:19.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-12-03T11:24:19.714 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2023-12-03T11:24:19.716 DEBUG:teuthology.parallel:result is None 2023-12-03T11:24:19.716 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-03T11:24:19.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:24:20.050 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:20.051 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-03T11:24:20.062 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-03T11:24:18.075769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218720,"kb_used_data":2556,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756000,"statfs":{"total":773094113280,"available":772870144000,"internally_reserved":0,"allocated":2617344,"data_stored":1633284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000832"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.117691+0000","last_change":"2023-12-03T11:24:16.116302+0000","last_active":"2023-12-03T11:24:16.117691+0000","last_peered":"2023-12-03T11:24:16.117691+0000","last_clean":"2023-12-03T11:24:16.117691+0000","last_became_active":"2023-12-03T11:24:16.115960+0000","last_became_peered":"2023-12-03T11:24:16.115960+0000","last_unstale":"2023-12-03T11:24:16.117691+0000","last_undegraded":"2023-12-03T11:24:16.117691+0000","last_fullsized":"2023-12-03T11:24:16.117691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T21:49:51.451813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.118575+0000","last_change":"2023-12-03T11:24:16.115194+0000","last_active":"2023-12-03T11:24:16.118575+0000","last_peered":"2023-12-03T11:24:16.118575+0000","last_clean":"2023-12-03T11:24:16.118575+0000","last_became_active":"2023-12-03T11:24:16.114827+0000","last_became_peered":"2023-12-03T11:24:16.114827+0000","last_unstale":"2023-12-03T11:24:16.118575+0000","last_undegraded":"2023-12-03T11:24:16.118575+0000","last_fullsized":"2023-12-03T11:24:16.118575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T17:11:23.703805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.292350+0000","last_change":"2023-12-03T11:24:16.291330+0000","last_active":"2023-12-03T11:24:16.292350+0000","last_peered":"2023-12-03T11:24:16.292350+0000","last_clean":"2023-12-03T11:24:16.292350+0000","last_became_active":"2023-12-03T11:24:16.291000+0000","last_became_peered":"2023-12-03T11:24:16.291000+0000","last_unstale":"2023-12-03T11:24:16.292350+0000","last_undegraded":"2023-12-03T11:24:16.292350+0000","last_fullsized":"2023-12-03T11:24:16.292350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:59:22.617538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.291199+0000","last_change":"2023-12-03T11:24:16.290200+0000","last_active":"2023-12-03T11:24:16.291199+0000","last_peered":"2023-12-03T11:24:16.291199+0000","last_clean":"2023-12-03T11:24:16.291199+0000","last_became_active":"2023-12-03T11:24:16.289841+0000","last_became_peered":"2023-12-03T11:24:16.289841+0000","last_unstale":"2023-12-03T11:24:16.291199+0000","last_undegraded":"2023-12-03T11:24:16.291199+0000","last_fullsized":"2023-12-03T11:24:16.291199+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:20:15.671109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.115903+0000","last_change":"2023-12-03T11:24:16.114819+0000","last_active":"2023-12-03T11:24:16.115903+0000","last_peered":"2023-12-03T11:24:16.115903+0000","last_clean":"2023-12-03T11:24:16.115903+0000","last_became_active":"2023-12-03T11:24:16.114494+0000","last_became_peered":"2023-12-03T11:24:16.114494+0000","last_unstale":"2023-12-03T11:24:16.115903+0000","last_undegraded":"2023-12-03T11:24:16.115903+0000","last_fullsized":"2023-12-03T11:24:16.115903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T23:00:10.454660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:24:17.114266+0000","last_change":"2023-12-03T11:24:16.350854+0000","last_active":"2023-12-03T11:24:17.114266+0000","last_peered":"2023-12-03T11:24:17.114266+0000","last_clean":"2023-12-03T11:24:17.114266+0000","last_became_active":"2023-12-03T11:24:16.350534+0000","last_became_peered":"2023-12-03T11:24:16.350534+0000","last_unstale":"2023-12-03T11:24:17.114266+0000","last_undegraded":"2023-12-03T11:24:17.114266+0000","last_fullsized":"2023-12-03T11:24:17.114266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:07:04.694544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.116689+0000","last_change":"2023-12-03T11:24:16.115485+0000","last_active":"2023-12-03T11:24:16.116689+0000","last_peered":"2023-12-03T11:24:16.116689+0000","last_clean":"2023-12-03T11:24:16.116689+0000","last_became_active":"2023-12-03T11:24:16.115074+0000","last_became_peered":"2023-12-03T11:24:16.115074+0000","last_unstale":"2023-12-03T11:24:16.116689+0000","last_undegraded":"2023-12-03T11:24:16.116689+0000","last_fullsized":"2023-12-03T11:24:16.116689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T11:41:18.066745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.115207+0000","last_change":"2023-12-03T11:24:16.114387+0000","last_active":"2023-12-03T11:24:16.115207+0000","last_peered":"2023-12-03T11:24:16.115207+0000","last_clean":"2023-12-03T11:24:16.115207+0000","last_became_active":"2023-12-03T11:24:16.114052+0000","last_became_peered":"2023-12-03T11:24:16.114052+0000","last_unstale":"2023-12-03T11:24:16.115207+0000","last_undegraded":"2023-12-03T11:24:16.115207+0000","last_fullsized":"2023-12-03T11:24:16.115207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:31:08.661492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":163,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-03T11:24:15.302633+0000","last_change":"2023-12-03T11:24:10.079199+0000","last_active":"2023-12-03T11:24:15.302633+0000","last_peered":"2023-12-03T11:24:15.302633+0000","last_clean":"2023-12-03T11:24:15.302633+0000","last_became_active":"2023-12-03T11:24:10.078189+0000","last_became_peered":"2023-12-03T11:24:10.078189+0000","last_unstale":"2023-12-03T11:24:15.302633+0000","last_undegraded":"2023-12-03T11:24:15.302633+0000","last_fullsized":"2023-12-03T11:24:15.302633+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:09.068387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:09.068387+0000","last_clean_scrub_stamp":"2023-12-03T11:24:09.068387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:41:15.025151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":51504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-03T11:24:20.063 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T11:24:20.360 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:20.361 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2023-12-03T11:24:20.374 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-03T11:24:20.076099+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218780,"kb_used_data":2604,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755940,"statfs":{"total":773094113280,"available":772870082560,"internally_reserved":0,"allocated":2666496,"data_stored":1656316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001163"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.117691+0000","last_change":"2023-12-03T11:24:16.116302+0000","last_active":"2023-12-03T11:24:16.117691+0000","last_peered":"2023-12-03T11:24:16.117691+0000","last_clean":"2023-12-03T11:24:16.117691+0000","last_became_active":"2023-12-03T11:24:16.115960+0000","last_became_peered":"2023-12-03T11:24:16.115960+0000","last_unstale":"2023-12-03T11:24:16.117691+0000","last_undegraded":"2023-12-03T11:24:16.117691+0000","last_fullsized":"2023-12-03T11:24:16.117691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T21:49:51.451813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:24:17.518274+0000","last_change":"2023-12-03T11:24:16.115194+0000","last_active":"2023-12-03T11:24:17.518274+0000","last_peered":"2023-12-03T11:24:17.518274+0000","last_clean":"2023-12-03T11:24:17.518274+0000","last_became_active":"2023-12-03T11:24:16.114827+0000","last_became_peered":"2023-12-03T11:24:16.114827+0000","last_unstale":"2023-12-03T11:24:17.518274+0000","last_undegraded":"2023-12-03T11:24:17.518274+0000","last_fullsized":"2023-12-03T11:24:17.518274+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T13:02:29.087500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.292350+0000","last_change":"2023-12-03T11:24:16.291330+0000","last_active":"2023-12-03T11:24:16.292350+0000","last_peered":"2023-12-03T11:24:16.292350+0000","last_clean":"2023-12-03T11:24:16.292350+0000","last_became_active":"2023-12-03T11:24:16.291000+0000","last_became_peered":"2023-12-03T11:24:16.291000+0000","last_unstale":"2023-12-03T11:24:16.292350+0000","last_undegraded":"2023-12-03T11:24:16.292350+0000","last_fullsized":"2023-12-03T11:24:16.292350+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:59:22.617538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:24:17.518245+0000","last_change":"2023-12-03T11:24:16.290200+0000","last_active":"2023-12-03T11:24:17.518245+0000","last_peered":"2023-12-03T11:24:17.518245+0000","last_clean":"2023-12-03T11:24:17.518245+0000","last_became_active":"2023-12-03T11:24:16.289841+0000","last_became_peered":"2023-12-03T11:24:16.289841+0000","last_unstale":"2023-12-03T11:24:17.518245+0000","last_undegraded":"2023-12-03T11:24:17.518245+0000","last_fullsized":"2023-12-03T11:24:17.518245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T13:09:54.424071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.115903+0000","last_change":"2023-12-03T11:24:16.114819+0000","last_active":"2023-12-03T11:24:16.115903+0000","last_peered":"2023-12-03T11:24:16.115903+0000","last_clean":"2023-12-03T11:24:16.115903+0000","last_became_active":"2023-12-03T11:24:16.114494+0000","last_became_peered":"2023-12-03T11:24:16.114494+0000","last_unstale":"2023-12-03T11:24:16.115903+0000","last_undegraded":"2023-12-03T11:24:16.115903+0000","last_fullsized":"2023-12-03T11:24:16.115903+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T23:00:10.454660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-12-03T11:24:17.114266+0000","last_change":"2023-12-03T11:24:16.350854+0000","last_active":"2023-12-03T11:24:17.114266+0000","last_peered":"2023-12-03T11:24:17.114266+0000","last_clean":"2023-12-03T11:24:17.114266+0000","last_became_active":"2023-12-03T11:24:16.350534+0000","last_became_peered":"2023-12-03T11:24:16.350534+0000","last_unstale":"2023-12-03T11:24:17.114266+0000","last_undegraded":"2023-12-03T11:24:17.114266+0000","last_fullsized":"2023-12-03T11:24:17.114266+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:07:04.694544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.116689+0000","last_change":"2023-12-03T11:24:16.115485+0000","last_active":"2023-12-03T11:24:16.116689+0000","last_peered":"2023-12-03T11:24:16.116689+0000","last_clean":"2023-12-03T11:24:16.116689+0000","last_became_active":"2023-12-03T11:24:16.115074+0000","last_became_peered":"2023-12-03T11:24:16.115074+0000","last_unstale":"2023-12-03T11:24:16.116689+0000","last_undegraded":"2023-12-03T11:24:16.116689+0000","last_fullsized":"2023-12-03T11:24:16.116689+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T11:41:18.066745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-12-03T11:24:16.115207+0000","last_change":"2023-12-03T11:24:16.114387+0000","last_active":"2023-12-03T11:24:16.115207+0000","last_peered":"2023-12-03T11:24:16.115207+0000","last_clean":"2023-12-03T11:24:16.115207+0000","last_became_active":"2023-12-03T11:24:16.114052+0000","last_became_peered":"2023-12-03T11:24:16.114052+0000","last_unstale":"2023-12-03T11:24:16.115207+0000","last_undegraded":"2023-12-03T11:24:16.115207+0000","last_fullsized":"2023-12-03T11:24:16.115207+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:15.093516+0000","last_clean_scrub_stamp":"2023-12-03T11:24:15.093516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T22:31:08.661492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":163,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-03T11:24:15.302633+0000","last_change":"2023-12-03T11:24:10.079199+0000","last_active":"2023-12-03T11:24:15.302633+0000","last_peered":"2023-12-03T11:24:15.302633+0000","last_clean":"2023-12-03T11:24:15.302633+0000","last_became_active":"2023-12-03T11:24:10.078189+0000","last_became_peered":"2023-12-03T11:24:10.078189+0000","last_unstale":"2023-12-03T11:24:15.302633+0000","last_undegraded":"2023-12-03T11:24:15.302633+0000","last_fullsized":"2023-12-03T11:24:15.302633+0000","mapping_epoch":14,"log_start":"15'100","ondisk_log_start":"15'100","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-03T11:24:09.068387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-03T11:24:09.068387+0000","last_clean_scrub_stamp":"2023-12-03T11:24:09.068387+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":100,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-04T19:41:15.025151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":63040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":180224,"data_stored":57292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6839999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92800000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":647660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-03T11:24:20.375 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-03T11:24:20.375 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-03T11:24:20.375 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-03T11:24:20.375 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-03T11:24:20.702 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:24:20.702 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-03T11:24:20.715 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-03T11:24:20.715 INFO:teuthology.run_tasks:Running task openssl_keys... 2023-12-03T11:24:20.726 INFO:teuthology.run_tasks:Running task rgw... 2023-12-03T11:24:20.740 DEBUG:tasks.rgw:config is {'client.0': None} 2023-12-03T11:24:20.740 DEBUG:tasks.rgw:client list is dict_keys(['client.0']) 2023-12-03T11:24:20.741 INFO:tasks.rgw:Creating data pools 2023-12-03T11:24:20.741 DEBUG:tasks.rgw:Obtaining remote for client client.0 2023-12-03T11:24:20.741 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd erasure-code-profile set client.0 crush-failure-domain=osd k=3 m=1 name=testprofile --cluster ceph 2023-12-03T11:24:21.131 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.data 64 64 erasure client.0 --cluster ceph 2023-12-03T11:24:23.127 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.data' created 2023-12-03T11:24:23.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster ceph 2023-12-03T11:24:24.134 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2023-12-03T11:24:24.168 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster ceph 2023-12-03T11:24:25.149 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.index' created 2023-12-03T11:24:25.176 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster ceph 2023-12-03T11:24:26.151 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2023-12-03T11:24:26.189 DEBUG:tasks.rgw:Pools created 2023-12-03T11:24:26.190 INFO:tasks.rgw:Configuring compression type = random 2023-12-03T11:24:26.190 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:24:26.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:24:26.190 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:24:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:26.288 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:26.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:26.299+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.180+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.180+0000 7f0ed433d800 20 realm 2023-12-03T11:24:28.182 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:24:28.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.181+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.182+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.182+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.183+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.183+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.183+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.183+0000 7f0ed433d800 10 Creating default zonegroup 2023-12-03T11:24:28.185 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.183+0000 7f0ed433d800 10 couldn't find old data placement pools config, setting up new ones for the zone 2023-12-03T11:24:28.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.196+0000 7f0ed00a3640 0 -- 172.21.15.12:0/3048167838 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f0ec807e040 msgr2=0x7f0ec8065990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.198+0000 7f0ed433d800 10 WARNING: driver->list_zones() returned r=-2 2023-12-03T11:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.198+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.198+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.202+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.203+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.203+0000 7f0ed433d800 10 could not read realm id: (2) No such file or directory 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.203+0000 7f0ed433d800 10 WARNING: failed to set zone as default, r=-22 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.203+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.203+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.206+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.207 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.206+0000 7f0ed433d800 20 rados_obj.operate() r=0 bl.length=874 2023-12-03T11:24:28.208 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.206+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.210+0000 7f0ed433d800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:24:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.210+0000 7f0ed433d800 10 Using default name default 2023-12-03T11:24:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.210+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.211 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:24:28.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados_obj.operate() r=0 bl.length=874 2023-12-03T11:24:28.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 zone default found 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 4 Realm: () 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 zonegroup default 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:28.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:28.211+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:29.170 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:29.168+0000 7f0ed00a3640 0 -- 172.21.15.12:0/3048167838 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x557da6a42d10 msgr2=0x557da6a497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:24:30.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:30.188+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:30.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:30.188+0000 7f0ed433d800 20 rados->read ofs=0 len=0 2023-12-03T11:24:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:30.194+0000 7f0ed433d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:30.194+0000 7f0ed433d800 20 started sync module instance, tier type = 2023-12-03T11:24:30.195 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:30.194+0000 7f0ed433d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=0 2023-12-03T11:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=1 2023-12-03T11:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=2 2023-12-03T11:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=3 2023-12-03T11:24:32.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=4 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=5 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.215+0000 7f0ed433d800 20 add_watcher() i=6 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.216+0000 7f0ed433d800 20 add_watcher() i=7 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.216+0000 7f0ed433d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.216+0000 7f0ed433d800 5 tl::expected, obj_version>, boost::system::error_code> logback_generations::read(const DoutPrefixProvider*, optional_yield):410: oid=data_loggenerations_metadata not found 2023-12-03T11:24:32.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.216+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.0 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.216+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.0 does not exist 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.1 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.1 does not exist 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.2 2023-12-03T11:24:32.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.217+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.2 does not exist 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.3 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.3 does not exist 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.4 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.4 does not exist 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.5 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.5 does not exist 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.6 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.218+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.6 does not exist 2023-12-03T11:24:32.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.7 2023-12-03T11:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.7 does not exist 2023-12-03T11:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.8 2023-12-03T11:24:32.221 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.8 does not exist 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.9 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.219+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.9 does not exist 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.10 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.10 does not exist 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.11 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.222 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.11 does not exist 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.12 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.12 does not exist 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.13 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.220+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.13 does not exist 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.14 2023-12-03T11:24:32.223 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.14 does not exist 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.15 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.15 does not exist 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.16 2023-12-03T11:24:32.227 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.16 does not exist 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.17 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.221+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.17 does not exist 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.18 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.18 does not exist 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.19 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.19 does not exist 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.20 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.222+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.20 does not exist 2023-12-03T11:24:32.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.21 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.21 does not exist 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.22 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.22 does not exist 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.23 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.23 does not exist 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.24 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.223+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.24 does not exist 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.25 2023-12-03T11:24:32.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.25 does not exist 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.26 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.26 does not exist 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.27 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.224+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.27 does not exist 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.28 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.28 does not exist 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.29 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.29 does not exist 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.30 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.30 does not exist 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.31 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.225+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.31 does not exist 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.32 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.32 does not exist 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.33 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.33 does not exist 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.34 2023-12-03T11:24:32.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.34 does not exist 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.35 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.226+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.35 does not exist 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.36 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.36 does not exist 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.37 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.37 does not exist 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.38 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.38 does not exist 2023-12-03T11:24:32.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.39 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.227+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.39 does not exist 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.40 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.40 does not exist 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.41 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.41 does not exist 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.42 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.42 does not exist 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.43 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.228+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.43 does not exist 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.44 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.44 does not exist 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.45 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.45 does not exist 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.46 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.46 does not exist 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.47 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.229+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.47 does not exist 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.48 2023-12-03T11:24:32.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.48 does not exist 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.49 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.49 does not exist 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.50 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.50 does not exist 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.51 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.230+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.51 does not exist 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.52 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.52 does not exist 2023-12-03T11:24:32.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.53 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.53 does not exist 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.54 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.54 does not exist 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.55 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.231+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.55 does not exist 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.56 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.56 does not exist 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.57 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.57 does not exist 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.58 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.58 does not exist 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.59 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.232+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.59 does not exist 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.60 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.60 does not exist 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.61 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.61 does not exist 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.62 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.62 does not exist 2023-12-03T11:24:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.63 2023-12-03T11:24:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.233+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.63 does not exist 2023-12-03T11:24:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.64 2023-12-03T11:24:32.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.64 does not exist 2023-12-03T11:24:32.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.65 2023-12-03T11:24:32.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.65 does not exist 2023-12-03T11:24:32.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.66 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.66 does not exist 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.67 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.67 does not exist 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.68 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.234+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.68 does not exist 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.69 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.69 does not exist 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.70 2023-12-03T11:24:32.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.70 does not exist 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.71 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.71 does not exist 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.72 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.235+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.72 does not exist 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.73 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.73 does not exist 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.74 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.74 does not exist 2023-12-03T11:24:32.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.75 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.75 does not exist 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.76 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.236+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.76 does not exist 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.77 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.77 does not exist 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.78 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.78 does not exist 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.79 2023-12-03T11:24:32.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.237+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.79 does not exist 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.80 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.80 does not exist 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.81 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.81 does not exist 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.82 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.82 does not exist 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.83 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.238+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.83 does not exist 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.84 2023-12-03T11:24:32.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.84 does not exist 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.85 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.85 does not exist 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.86 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.86 does not exist 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.87 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.239+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.87 does not exist 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.88 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.88 does not exist 2023-12-03T11:24:32.250 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.89 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.89 does not exist 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.90 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.240+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.90 does not exist 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.91 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.91 does not exist 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.92 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.92 does not exist 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.93 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.93 does not exist 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.94 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.241+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.94 does not exist 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.95 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.95 does not exist 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.96 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.96 does not exist 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.97 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.97 does not exist 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.98 2023-12-03T11:24:32.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.242+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.98 does not exist 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.99 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.99 does not exist 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.100 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.100 does not exist 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.101 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.101 does not exist 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.102 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.243+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.102 does not exist 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.103 2023-12-03T11:24:32.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.103 does not exist 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.104 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.104 does not exist 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.105 2023-12-03T11:24:32.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.105 does not exist 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.106 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.244+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.106 does not exist 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.107 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.107 does not exist 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.108 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.108 does not exist 2023-12-03T11:24:32.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.109 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.109 does not exist 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.110 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.245+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.110 does not exist 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.111 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.111 does not exist 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.112 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.112 does not exist 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.113 2023-12-03T11:24:32.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.113 does not exist 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.114 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.246+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.114 does not exist 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.115 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.115 does not exist 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.116 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.116 does not exist 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.117 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.247+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.117 does not exist 2023-12-03T11:24:32.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.118 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.118 does not exist 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.119 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.119 does not exist 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.120 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.120 does not exist 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.121 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.248+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.121 does not exist 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.122 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.122 does not exist 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.123 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.123 does not exist 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.124 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.124 does not exist 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.125 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.249+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.125 does not exist 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.126 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.126 does not exist 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):38 probing oid=data_log.127 2023-12-03T11:24:32.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 shard_check {anonymous}::probe_shard(const DoutPrefixProvider*, librados::v14_2_0::IoCtx&, const string&, bool&, optional_yield):60: oid=data_log.127 does not exist 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.250+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::create(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, std::optional >, bool, uint64_t, uint64_t):1210 entering 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.252+0000 7f0ed433d800 20 static int rgw::cls::fifo::FIFO::open(const DoutPrefixProvider*, librados::v14_2_0::IoCtx, std::string, std::unique_ptr*, optional_yield, std::optional, bool):1168 entering 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.255+0000 7f0e967e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.255+0000 7f0ed433d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:32.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:32.255+0000 7f0ed433d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.200 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.199+0000 7f0ed433d800 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2023-12-03T11:24:35.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.199+0000 7f0ed433d800 5 note: GC not initialized 2023-12-03T11:24:35.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.200+0000 7f0dfd7fa640 20 reqs_thread_entry: start 2023-12-03T11:24:35.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.200+0000 7f0ed433d800 20 init_complete bucket index max shards: 1 2023-12-03T11:24:35.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.200+0000 7f0ed433d800 20 Filter name: none 2023-12-03T11:24:35.201 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.200+0000 7f0deeffd640 20 reqs_thread_entry: start 2023-12-03T11:24:35.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.230+0000 7f0e967e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.233+0000 7f0ed433d800 20 remove_watcher() i=0 2023-12-03T11:24:35.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.233+0000 7f0ed433d800 2 removed watcher, disabling cache 2023-12-03T11:24:35.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.235+0000 7f0ed433d800 20 remove_watcher() i=1 2023-12-03T11:24:35.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.236+0000 7f0ed433d800 20 remove_watcher() i=2 2023-12-03T11:24:35.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.240+0000 7f0ed433d800 20 remove_watcher() i=3 2023-12-03T11:24:35.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.242+0000 7f0ed433d800 20 remove_watcher() i=4 2023-12-03T11:24:35.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.243+0000 7f0ed433d800 20 remove_watcher() i=5 2023-12-03T11:24:35.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.244+0000 7f0ed433d800 20 remove_watcher() i=6 2023-12-03T11:24:35.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.246+0000 7f0ed433d800 20 remove_watcher() i=7 2023-12-03T11:24:35.258 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2023-12-03T11:24:35.258 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:24:35.258 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-12-03T11:24:35.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2023-12-03T11:24:35.259 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --compression random 2023-12-03T11:24:35.310 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.310 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.322+0000 7fa0bb72a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.323 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.322+0000 7fa0bb72a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.325+0000 7fa0257fa640 20 reqs_thread_entry: start 2023-12-03T11:24:35.338 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"ca5e6929-62e3-48b1-b1be-09f9f5399756","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":true}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:24:35.338 DEBUG:tasks.util.rgw: json result: {'id': 'ca5e6929-62e3-48b1-b1be-09f9f5399756', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': True}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:24:35.338 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:24:35.339 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:24:35.339 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:24:35.381 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.382 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.392+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.394+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.394+0000 7f4595cba800 20 realm 2023-12-03T11:24:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.394+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.395+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.395+0000 7f4595cba800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:24:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.395+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.395+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.395+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.396+0000 7f4595cba800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:24:35.397 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.396+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.397+0000 7f4595cba800 20 rados_obj.operate() r=0 bl.length=890 2023-12-03T11:24:35.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.397+0000 7f4595cba800 20 searching for the correct realm 2023-12-03T11:24:35.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.406+0000 7f45937ba640 0 -- 172.21.15.12:0/2707855294 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f4588061070 msgr2=0x7f45880972e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:24:35.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:24:35.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:24:35.409 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:24:35.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:24:35.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.408+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 zone default found 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 4 Realm: () 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:24:35.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 zonegroup default 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.409+0000 7f4595cba800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.410+0000 7f4595cba800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.410+0000 7f4595cba800 20 started sync module instance, tier type = 2023-12-03T11:24:35.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.410+0000 7f4595cba800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:24:35.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.414+0000 7f4592fb9640 0 -- 172.21.15.12:0/2707855294 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x56323b3b62d0 msgr2=0x56323b3b86c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=0 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=1 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=2 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=3 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=4 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=5 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.421+0000 7f4595cba800 20 add_watcher() i=6 2023-12-03T11:24:35.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.422+0000 7f4595cba800 20 add_watcher() i=7 2023-12-03T11:24:35.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.422+0000 7f4595cba800 2 all 8 watchers are set, enabling cache 2023-12-03T11:24:35.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.424+0000 7f45597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:35.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.424+0000 7f4595cba800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.424+0000 7f4595cba800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.429+0000 7f4595cba800 5 note: GC not initialized 2023-12-03T11:24:35.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.429+0000 7f44c7fff640 20 reqs_thread_entry: start 2023-12-03T11:24:35.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.429+0000 7f4595cba800 20 init_complete bucket index max shards: 1 2023-12-03T11:24:35.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.429+0000 7f4595cba800 20 Filter name: none 2023-12-03T11:24:35.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.429+0000 7f44c5ffb640 20 reqs_thread_entry: start 2023-12-03T11:24:35.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.440+0000 7f45597e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:35.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.443+0000 7f4595cba800 20 remove_watcher() i=0 2023-12-03T11:24:35.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.443+0000 7f4595cba800 2 removed watcher, disabling cache 2023-12-03T11:24:35.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.444+0000 7f4595cba800 20 remove_watcher() i=1 2023-12-03T11:24:35.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.445+0000 7f4595cba800 20 remove_watcher() i=2 2023-12-03T11:24:35.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.447+0000 7f4595cba800 20 remove_watcher() i=3 2023-12-03T11:24:35.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.448+0000 7f4595cba800 20 remove_watcher() i=4 2023-12-03T11:24:35.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.449+0000 7f4595cba800 20 remove_watcher() i=5 2023-12-03T11:24:35.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.450+0000 7f4595cba800 20 remove_watcher() i=6 2023-12-03T11:24:35.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.451+0000 7f4595cba800 20 remove_watcher() i=7 2023-12-03T11:24:35.457 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2023-12-03T11:24:35.457 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:24:35.457 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2023-12-03T11:24:35.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--placement-inline-data', 'false'] 2023-12-03T11:24:35.457 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement modify --rgw-zone default --placement-id default-placement --placement-inline-data false 2023-12-03T11:24:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.545 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.556+0000 7fa18b128800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.556+0000 7fa18b128800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.559+0000 7fa0bffff640 20 reqs_thread_entry: start 2023-12-03T11:24:35.571 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"ca5e6929-62e3-48b1-b1be-09f9f5399756","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:24:35.571 DEBUG:tasks.util.rgw: json result: {'id': 'ca5e6929-62e3-48b1-b1be-09f9f5399756', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:24:35.571 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'list'] 2023-12-03T11:24:35.571 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'user', 'list'] 2023-12-03T11:24:35.571 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph user list 2023-12-03T11:24:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.613 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.623+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 realm 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.627 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.625+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.626+0000 7fdddb722800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:24:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.626+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.627+0000 7fdddb722800 20 rados_obj.operate() r=0 bl.length=890 2023-12-03T11:24:35.628 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.627+0000 7fdddb722800 20 searching for the correct realm 2023-12-03T11:24:35.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.637+0000 7fddd7370640 0 -- 172.21.15.12:0/3619559926 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5653c1dc84a0 msgr2=0x5653c1dca8f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:24:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:24:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:24:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:24:35.639 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:24:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.640 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:24:35.641 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.638+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados_obj.operate() r=0 bl.length=420 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 zone default found 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 4 Realm: () 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:24:35.642 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 zonegroup default 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados->read ofs=0 len=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.639+0000 7fdddb722800 20 started sync module instance, tier type = 2023-12-03T11:24:35.643 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.640+0000 7fdddb722800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:24:35.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.650+0000 7fddd7b71640 0 -- 172.21.15.12:0/3619559926 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5653c1dcb3c0 msgr2=0x5653c1dd2a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:24:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=0 2023-12-03T11:24:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=1 2023-12-03T11:24:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=2 2023-12-03T11:24:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=3 2023-12-03T11:24:35.655 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=4 2023-12-03T11:24:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=5 2023-12-03T11:24:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.653+0000 7fdddb722800 20 add_watcher() i=6 2023-12-03T11:24:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.654+0000 7fdddb722800 20 add_watcher() i=7 2023-12-03T11:24:35.656 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.654+0000 7fdddb722800 2 all 8 watchers are set, enabling cache 2023-12-03T11:24:35.657 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.656+0000 7fdd9d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.656+0000 7fdddb722800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.658 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.656+0000 7fdddb722800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.661+0000 7fdddb722800 5 note: GC not initialized 2023-12-03T11:24:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.661+0000 7fdd0bfff640 20 reqs_thread_entry: start 2023-12-03T11:24:35.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.662+0000 7fdddb722800 20 init_complete bucket index max shards: 1 2023-12-03T11:24:35.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.662+0000 7fdddb722800 20 Filter name: none 2023-12-03T11:24:35.664 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.662+0000 7fdd09ffb640 20 reqs_thread_entry: start 2023-12-03T11:24:35.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.672+0000 7fdd9d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:24:35.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.674+0000 7fdddb722800 20 remove_watcher() i=0 2023-12-03T11:24:35.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.674+0000 7fdddb722800 2 removed watcher, disabling cache 2023-12-03T11:24:35.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.675+0000 7fdddb722800 20 remove_watcher() i=1 2023-12-03T11:24:35.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.677+0000 7fdddb722800 20 remove_watcher() i=2 2023-12-03T11:24:35.679 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.678+0000 7fdddb722800 20 remove_watcher() i=3 2023-12-03T11:24:35.680 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.679+0000 7fdddb722800 20 remove_watcher() i=4 2023-12-03T11:24:35.681 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.680+0000 7fdddb722800 20 remove_watcher() i=5 2023-12-03T11:24:35.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.682+0000 7fdddb722800 20 remove_watcher() i=6 2023-12-03T11:24:35.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.683+0000 7fdddb722800 20 remove_watcher() i=7 2023-12-03T11:24:35.689 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2023-12-03T11:24:35.689 DEBUG:tasks.util.rgw: json result: [] 2023-12-03T11:24:35.689 INFO:tasks.rgw:Configuring storage class type = LUKEWARM 2023-12-03T11:24:35.689 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-12-03T11:24:35.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM'] 2023-12-03T11:24:35.689 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM 2023-12-03T11:24:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.779 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.790+0000 7fe547b3d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.791 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.790+0000 7fe547b3d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.794 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.793+0000 7fe481ffb640 20 reqs_thread_entry: start 2023-12-03T11:24:35.803 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["LUKEWARM","STANDARD"]}}] 2023-12-03T11:24:35.803 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['LUKEWARM', 'STANDARD']}}] 2023-12-03T11:24:35.803 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-12-03T11:24:35.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'LUKEWARM', '--data-pool', 'default.rgw.buckets.data.lukewarm'] 2023-12-03T11:24:35.804 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class LUKEWARM --data-pool default.rgw.buckets.data.lukewarm 2023-12-03T11:24:35.853 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.854 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.864+0000 7f1c2320f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.865 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.864+0000 7f1c2320f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.868 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.867+0000 7f1b5bfff640 20 reqs_thread_entry: start 2023-12-03T11:24:35.879 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"ca5e6929-62e3-48b1-b1be-09f9f5399756","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:24:35.879 DEBUG:tasks.util.rgw: json result: {'id': 'ca5e6929-62e3-48b1-b1be-09f9f5399756', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:24:35.880 INFO:tasks.rgw:Configuring storage class type = FROZEN 2023-12-03T11:24:35.880 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-12-03T11:24:35.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zonegroup', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN'] 2023-12-03T11:24:35.880 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zonegroup placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN 2023-12-03T11:24:35.927 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:35.938 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.937+0000 7f99a972a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:35.939 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.937+0000 7f99a972a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:35.942 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:35.941+0000 7f98e2ffd640 20 reqs_thread_entry: start 2023-12-03T11:24:35.952 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":"default-placement","val":{"name":"default-placement","tags":[],"storage_classes":["FROZEN","LUKEWARM","STANDARD"]}}] 2023-12-03T11:24:35.952 DEBUG:tasks.util.rgw: json result: [{'key': 'default-placement', 'val': {'name': 'default-placement', 'tags': [], 'storage_classes': ['FROZEN', 'LUKEWARM', 'STANDARD']}}] 2023-12-03T11:24:35.952 INFO:tasks.util.rgw:rgwadmin: client.0 : ['zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-12-03T11:24:35.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'zone', 'placement', 'add', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--storage-class', 'FROZEN', '--data-pool', 'default.rgw.buckets.data.frozen'] 2023-12-03T11:24:35.952 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph zone placement add --rgw-zone default --placement-id default-placement --storage-class FROZEN --data-pool default.rgw.buckets.data.frozen 2023-12-03T11:24:36.002 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:24:36.003 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:24:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:36.013+0000 7ff58c33d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:24:36.014 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:36.013+0000 7ff58c33d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:24:36.017 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:24:36.016+0000 7ff4c67fc640 20 reqs_thread_entry: start 2023-12-03T11:24:36.027 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"ca5e6929-62e3-48b1-b1be-09f9f5399756","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"FROZEN":{"data_pool":"default.rgw.buckets.data.frozen"},"LUKEWARM":{"data_pool":"default.rgw.buckets.data.lukewarm"},"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0,"inline_data":false}}],"realm_id":"","notif_pool":"default.rgw.log:notif"} 2023-12-03T11:24:36.028 DEBUG:tasks.util.rgw: json result: {'id': 'ca5e6929-62e3-48b1-b1be-09f9f5399756', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'FROZEN': {'data_pool': 'default.rgw.buckets.data.frozen'}, 'LUKEWARM': {'data_pool': 'default.rgw.buckets.data.lukewarm'}, 'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0, 'inline_data': False}}], 'realm_id': '', 'notif_pool': 'default.rgw.log:notif'} 2023-12-03T11:24:36.028 INFO:tasks.rgw:Starting rgw... 2023-12-03T11:24:36.028 INFO:tasks.rgw:rgw client.0 config is {} 2023-12-03T11:24:36.028 INFO:tasks.rgw:Using beast as radosgw frontend 2023-12-03T11:24:36.028 INFO:tasks.rgw.client.0:Restarting daemon 2023-12-03T11:24:36.028 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=80' -n client.0 --cluster ceph -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.ceph.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.ceph.client.0.stdout 2>&1 2023-12-03T11:24:36.030 INFO:tasks.rgw.client.0:Started 2023-12-03T11:24:36.030 INFO:tasks.rgw:Polling client.0 until it starts accepting connections on http://smithi012.front.sepia.ceph.com:80/ 2023-12-03T11:24:36.031 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:80/ 2023-12-03T11:24:36.062 DEBUG:teuthology.orchestra.run:got remote process result: 7 2023-12-03T11:24:36.063 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-03T11:24:36.063 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2023-12-03T11:24:36.063 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 2023-12-03T11:24:36.063 INFO:teuthology.orchestra.run.smithi012.stderr:curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 80: Connection refused 2023-12-03T11:24:37.064 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:80/ 2023-12-03T11:24:37.085 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2023-12-03T11:24:37.085 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2023-12-03T11:24:37.086 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 208k 0 --:--:-- --:--:-- --:--:-- 208k 2023-12-03T11:24:37.087 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2023-12-03T11:24:37.087 INFO:teuthology.run_tasks:Running task tox... 2023-12-03T11:24:37.097 INFO:tasks.tox:Deploying tox from pip... 2023-12-03T11:24:37.098 DEBUG:teuthology.orchestra.run.smithi012:> python3 -m venv /home/ubuntu/cephtest/tox-venv 2023-12-03T11:24:39.572 DEBUG:teuthology.orchestra.run.smithi012:> source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==3.15.0 2023-12-03T11:24:41.089 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting tox==3.15.0 2023-12-03T11:24:42.111 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading tox-3.15.0-py2.py3-none-any.whl (82 kB) 2023-12-03T11:24:43.188 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting toml>=0.9.4 2023-12-03T11:24:43.488 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2023-12-03T11:24:43.901 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting pluggy<1,>=0.12.0 2023-12-03T11:24:44.230 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB) 2023-12-03T11:24:44.898 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 2023-12-03T11:24:45.231 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading virtualenv-20.25.0-py3-none-any.whl (3.8 MB) 2023-12-03T11:24:53.397 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting filelock<4,>=3.0.0 2023-12-03T11:24:53.718 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading filelock-3.13.1-py3-none-any.whl (11 kB) 2023-12-03T11:24:54.127 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting packaging>=14 2023-12-03T11:24:54.453 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading packaging-23.2-py3-none-any.whl (53 kB) 2023-12-03T11:24:54.824 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting six<2,>=1.14.0 2023-12-03T11:24:55.148 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:24:55.572 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting py<2,>=1.4.17 2023-12-03T11:24:55.893 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading py-1.11.0-py2.py3-none-any.whl (98 kB) 2023-12-03T11:24:56.370 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting distlib<1,>=0.3.7 2023-12-03T11:24:56.691 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading distlib-0.3.7-py2.py3-none-any.whl (468 kB) 2023-12-03T11:24:57.542 INFO:teuthology.orchestra.run.smithi012.stdout:Collecting platformdirs<5,>=3.9.1 2023-12-03T11:24:57.894 INFO:teuthology.orchestra.run.smithi012.stdout: Downloading platformdirs-4.0.0-py3-none-any.whl (17 kB) 2023-12-03T11:24:58.039 INFO:teuthology.orchestra.run.smithi012.stdout:Installing collected packages: platformdirs, filelock, distlib, virtualenv, toml, six, py, pluggy, packaging, tox 2023-12-03T11:24:58.900 INFO:teuthology.orchestra.run.smithi012.stdout:Successfully installed distlib-0.3.7 filelock-3.13.1 packaging-23.2 platformdirs-4.0.0 pluggy-0.13.1 py-1.11.0 six-1.16.0 toml-0.10.2 tox-3.15.0 virtualenv-20.25.0 2023-12-03T11:25:00.097 INFO:teuthology.orchestra.run.smithi012.stderr:WARNING: You are using pip version 21.2.3; however, version 23.3.1 is available. 2023-12-03T11:25:00.098 INFO:teuthology.orchestra.run.smithi012.stderr:You should consider upgrading via the '/home/ubuntu/cephtest/tox-venv/bin/python3 -m pip install --upgrade pip' command. 2023-12-03T11:25:00.165 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:25:00.183 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:25:00.183 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:25:00.184 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:25:00.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:25:00.199 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:25:00.199 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:25:00.255 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:25:00.255 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:25:00.310 INFO:tasks.workunit:timeout=3h 2023-12-03T11:25:00.310 INFO:tasks.workunit:cleanup=True 2023-12-03T11:25:00.310 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:25:00.366 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:26:05.130 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 96% (11720/12123) Updating files: 97% (11760/12123) Updating files: 98% (11881/12123) Updating files: 99% (12002/12123) Updating files: 100% (12123/12123) Updating files: 100% (12123/12123), done. 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:26:05.308 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:26:05.311 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:26:05.376 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:26:05.378 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:26:05.379 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:26:06.010 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:26:06.078 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:26:06.121 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:26:06.123 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:26:06.123 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:26:06.164 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:26:06.167 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:26:06.167 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:26:06.224 INFO:tasks.workunit:Running workunits matching rgw/run-bucket-check.sh on client.0... 2023-12-03T11:26:06.226 INFO:tasks.workunit:Running workunit rgw/run-bucket-check.sh... 2023-12-03T11:26:06.226 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/run-bucket-check.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-12-03T11:26:06.284 INFO:tasks.workunit.client.0.smithi012.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-bucket-check.sh 2023-12-03T11:26:06.284 INFO:tasks.workunit.client.0.smithi012.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:26:06.284 INFO:tasks.workunit.client.0.smithi012.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ deactivate nondestructive 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ hash -r 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset VIRTUAL_ENV 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '!' nondestructive = nondestructive ']' 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ export VIRTUAL_ENV 2023-12-03T11:26:08.717 INFO:tasks.workunit.client.0.smithi012.stderr:++ _OLD_VIRTUAL_PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:26:08.718 INFO:tasks.workunit.client.0.smithi012.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ export PATH 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ _OLD_VIRTUAL_PS1= 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ PS1='(rgw) ' 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ export PS1 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:++ hash -r 2023-12-03T11:26:08.719 INFO:tasks.workunit.client.0.smithi012.stderr:+ pip install pip --upgrade 2023-12-03T11:26:08.959 INFO:tasks.workunit.client.0.smithi012.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:26:09.171 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting pip 2023-12-03T11:26:09.278 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:26:09.472 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: pip 2023-12-03T11:26:09.472 INFO:tasks.workunit.client.0.smithi012.stdout: Attempting uninstall: pip 2023-12-03T11:26:09.472 INFO:tasks.workunit.client.0.smithi012.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:26:09.636 INFO:tasks.workunit.client.0.smithi012.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:26:09.642 INFO:tasks.workunit.client.0.smithi012.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:26:10.655 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed pip-23.3.1 2023-12-03T11:26:10.908 INFO:tasks.workunit.client.0.smithi012.stderr:+ pip install boto3 2023-12-03T11:26:12.973 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting boto3 2023-12-03T11:26:13.061 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:26:14.327 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:26:14.357 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:26:14.681 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:26:14.702 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:26:15.019 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:26:15.046 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:26:15.340 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:26:15.366 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:26:15.542 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 1.4 MB/s eta 0:00:00 2023-12-03T11:26:15.852 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:26:15.871 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:26:15.926 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 733.8 kB/s eta 0:00:00 2023-12-03T11:26:16.254 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:26:16.271 INFO:tasks.workunit.client.0.smithi012.stdout: Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:26:16.360 INFO:tasks.workunit.client.0.smithi012.stdout:Downloading boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:26:16.419 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 2.3 MB/s eta 0:00:00 2023-12-03T11:26:16.440 INFO:tasks.workunit.client.0.smithi012.stdout:Downloading botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:26:17.005 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 22.4 MB/s eta 0:00:00 2023-12-03T11:26:17.024 INFO:tasks.workunit.client.0.smithi012.stdout:Downloading s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:26:17.078 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 kB 1.4 MB/s eta 0:00:00 2023-12-03T11:26:17.095 INFO:tasks.workunit.client.0.smithi012.stdout:Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:26:17.137 INFO:tasks.workunit.client.0.smithi012.stdout: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.8/143.8 kB 3.3 MB/s eta 0:00:00 2023-12-03T11:26:17.241 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:26:18.322 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:26:18.449 INFO:tasks.workunit.client.0.smithi012.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_bucket_check.py 2023-12-03T11:26:18.586 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin user create --uid check-tester --display-name "Check Testing" --access-key OJODXSLNX4LUNHQG99PA --secret 3l6ffld34qaymfomuh832j94738aie2x4p2o8h6n 2023-12-03T11:26:18.621 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:18.621 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:18.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.634+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 20 realm 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.636+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.637+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.637+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.638+0000 7f265af23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:18.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.638+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.639+0000 7f265af23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:18.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.639+0000 7f265af23800 20 searching for the correct realm 2023-12-03T11:26:18.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.650+0000 7f2656b43640 0 -- 172.21.15.12:0/871971127 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55fa68c788b0 msgr2=0x55fa68c7ad00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:18.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.651+0000 7f265af23800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.651+0000 7f265af23800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.651+0000 7f265af23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.651+0000 7f265af23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.651+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:18.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:18.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 zone default found 2023-12-03T11:26:18.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 4 Realm: () 2023-12-03T11:26:18.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:18.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:18.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 zonegroup default 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.652+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f2657344640 0 -- 172.21.15.12:0/871971127 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55fa68c809f0 msgr2=0x55fa68c82eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:18.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 started sync module instance, tier type = 2023-12-03T11:26:18.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.653+0000 7f265af23800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=0 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=1 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=2 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=3 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=4 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=5 2023-12-03T11:26:18.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.666+0000 7f265af23800 20 add_watcher() i=6 2023-12-03T11:26:18.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.667+0000 7f265af23800 20 add_watcher() i=7 2023-12-03T11:26:18.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.667+0000 7f265af23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.669+0000 7f2597fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.669+0000 7f265af23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:18.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.669+0000 7f265af23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:18.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.675+0000 7f265af23800 5 note: GC not initialized 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.675+0000 7f2586ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.675+0000 7f265af23800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.675+0000 7f265af23800 20 Filter name: none 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f2584ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache get: name=default.rgw.meta+users.uid+check-tester : miss 2023-12-03T11:26:18.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 adding default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : miss 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 20 rados->read ofs=0 len=0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x0 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 adding default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-12-03T11:26:18.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.676+0000 7f265af23800 10 cache get: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA : hit (negative entry) 2023-12-03T11:26:18.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.680+0000 7f265af23800 10 cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-12-03T11:26:18.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.680+0000 7f265af23800 10 moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:26:18.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.680+0000 7f265af23800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:check-tester, ofs0, ns] 2023-12-03T11:26:18.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.681+0000 7f263cff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822656 cookie 94533988126608 notifier 4574 bl.length()=559 2023-12-03T11:26:18.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.681+0000 7f263cff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+check-tester info.flags=0x17 2023-12-03T11:26:18.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.681+0000 7f263cff9640 10 rgw watcher librados: moving default.rgw.meta+users.uid+check-tester to cache LRU end 2023-12-03T11:26:18.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f265af23800 10 cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-12-03T11:26:18.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f265af23800 10 moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:26:18.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f265af23800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 0, obj: default.rgw.meta:users.keys:OJODXSLNX4LUNHQG99PA, ofs0, ns] 2023-12-03T11:26:18.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f263cff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 154618822656 cookie 139802461081600 notifier 4574 bl.length()=186 2023-12-03T11:26:18.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f263cff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA info.flags=0x7 2023-12-03T11:26:18.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.683+0000 7f263cff9640 10 rgw watcher librados: moving default.rgw.meta+users.keys+OJODXSLNX4LUNHQG99PA to cache LRU end 2023-12-03T11:26:18.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.685+0000 7f2597fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:18.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.687+0000 7f265af23800 20 remove_watcher() i=0 2023-12-03T11:26:18.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.687+0000 7f265af23800 2 removed watcher, disabling cache 2023-12-03T11:26:18.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.688+0000 7f265af23800 20 remove_watcher() i=1 2023-12-03T11:26:18.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.689+0000 7f265af23800 20 remove_watcher() i=2 2023-12-03T11:26:18.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.691+0000 7f265af23800 20 remove_watcher() i=3 2023-12-03T11:26:18.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.692+0000 7f265af23800 20 remove_watcher() i=4 2023-12-03T11:26:18.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.693+0000 7f265af23800 20 remove_watcher() i=5 2023-12-03T11:26:18.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.694+0000 7f265af23800 20 remove_watcher() i=6 2023-12-03T11:26:18.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:18.695+0000 7f265af23800 20 remove_watcher() i=7 2023-12-03T11:26:19.027 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: recalculated bucket check stats are correct 2023-12-03T11:26:19.027 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:19.028 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2023-12-03T11:26:19.057 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:19.057 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:19.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.070+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 realm 2023-12-03T11:26:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.072+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.073+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.073+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.074+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:19.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.074+0000 7f4c7d12f800 20 searching for the correct realm 2023-12-03T11:26:19.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.083+0000 7f4c78e68640 0 -- 172.21.15.12:0/566636156 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a9fc19afa0 msgr2=0x55a9fc19d3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.085+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 zone default found 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 4 Realm: () 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 zonegroup default 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 started sync module instance, tier type = 2023-12-03T11:26:19.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.086+0000 7f4c7d12f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:19.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.099+0000 7f4c79669640 0 -- 172.21.15.12:0/566636156 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55a9fc1a3020 msgr2=0x55a9fc1a5470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=0 2023-12-03T11:26:19.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=1 2023-12-03T11:26:19.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=2 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=3 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=4 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=5 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.102+0000 7f4c7d12f800 20 add_watcher() i=6 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.103+0000 7f4c7d12f800 20 add_watcher() i=7 2023-12-03T11:26:19.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.103+0000 7f4c7d12f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:19.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.105+0000 7f4c3f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.105+0000 7f4c7d12f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:19.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.105+0000 7f4c7d12f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:19.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.111+0000 7f4c7d12f800 5 note: GC not initialized 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.111+0000 7f4bae7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.111+0000 7f4c7d12f800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.111+0000 7f4c7d12f800 20 Filter name: none 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.111+0000 7f4b9ffff640 20 reqs_thread_entry: start 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:19.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:19.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.112+0000 7f4c7d12f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:26:19.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.113+0000 7f4c7d12f800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:26:19.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.116+0000 7f4c3f7de640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.118+0000 7f4c7d12f800 20 remove_watcher() i=0 2023-12-03T11:26:19.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.118+0000 7f4c7d12f800 2 removed watcher, disabling cache 2023-12-03T11:26:19.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.119+0000 7f4c7d12f800 20 remove_watcher() i=1 2023-12-03T11:26:19.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.121+0000 7f4c7d12f800 20 remove_watcher() i=2 2023-12-03T11:26:19.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.122+0000 7f4c7d12f800 20 remove_watcher() i=3 2023-12-03T11:26:19.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.123+0000 7f4c7d12f800 20 remove_watcher() i=4 2023-12-03T11:26:19.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.124+0000 7f4c7d12f800 20 remove_watcher() i=5 2023-12-03T11:26:19.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.125+0000 7f4c7d12f800 20 remove_watcher() i=6 2023-12-03T11:26:19.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.126+0000 7f4c7d12f800 20 remove_watcher() i=7 2023-12-03T11:26:19.133 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket check-bucket 2023-12-03T11:26:19.160 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:19.160 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:19.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.174+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.175+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.175+0000 7f662f011800 20 realm 2023-12-03T11:26:19.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.175+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.176+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.176+0000 7f662f011800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:19.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.176+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.176+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.176+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.177+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.177+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.178+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:19.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.178+0000 7f662f011800 20 searching for the correct realm 2023-12-03T11:26:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662a5b1640 0 -- 172.21.15.12:0/3209902406 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a37de8c690 msgr2=0x55a37de8ea80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.188+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:19.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 zone default found 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 4 Realm: () 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 zonegroup default 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.189+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.190+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.190+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.190+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.190+0000 7f662f011800 20 started sync module instance, tier type = 2023-12-03T11:26:19.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.190+0000 7f662f011800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:19.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.201+0000 7f662adb2640 0 -- 172.21.15.12:0/3209902406 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f6620099520 msgr2=0x7f6620066a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=0 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=1 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=2 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=3 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=4 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=5 2023-12-03T11:26:19.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.202+0000 7f662f011800 20 add_watcher() i=6 2023-12-03T11:26:19.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.203+0000 7f662f011800 20 add_watcher() i=7 2023-12-03T11:26:19.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.203+0000 7f662f011800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.206+0000 7f65effe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.206+0000 7f662f011800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:19.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.206+0000 7f662f011800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.212+0000 7f662f011800 5 note: GC not initialized 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f655effd640 20 reqs_thread_entry: start 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f662f011800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f662f011800 20 Filter name: none 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f655cff9640 20 reqs_thread_entry: start 2023-12-03T11:26:19.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:19.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.213+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:19.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:19.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:19.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:19.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.214+0000 7f662f011800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:19.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.215+0000 7f662f011800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.215+0000 7f662f011800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.216+0000 7f65effe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.220+0000 7f662f011800 20 remove_watcher() i=0 2023-12-03T11:26:19.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.220+0000 7f662f011800 2 removed watcher, disabling cache 2023-12-03T11:26:19.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.221+0000 7f662f011800 20 remove_watcher() i=1 2023-12-03T11:26:19.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.222+0000 7f662f011800 20 remove_watcher() i=2 2023-12-03T11:26:19.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.223+0000 7f662f011800 20 remove_watcher() i=3 2023-12-03T11:26:19.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.224+0000 7f662f011800 20 remove_watcher() i=4 2023-12-03T11:26:19.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.225+0000 7f662f011800 20 remove_watcher() i=5 2023-12-03T11:26:19.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.226+0000 7f662f011800 20 remove_watcher() i=6 2023-12-03T11:26:19.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.227+0000 7f662f011800 20 remove_watcher() i=7 2023-12-03T11:26:19.233 INFO:tasks.workunit.client.0.smithi012.stderr:{'rgw.main': {'size': 54, 'size_actual': 24576, 'size_utilized': 97, 'size_kb': 1, 'size_kb_actual': 24, 'size_kb_utilized': 1, 'num_objects': 6}} 2023-12-03T11:26:19.233 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check unlinked does not report normal entries 2023-12-03T11:26:19.233 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:19.234 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:26:19.266 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:19.266 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:19.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.280+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 realm 2023-12-03T11:26:19.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.282+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.283+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.283+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.284+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:19.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.284+0000 7fac0c6e4800 20 searching for the correct realm 2023-12-03T11:26:19.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.291+0000 7fac0a9e5640 0 -- 172.21.15.12:0/311149769 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x55a84750eb60 msgr2=0x55a847510f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:19.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.294+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 zone default found 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 4 Realm: () 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 zonegroup default 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.295+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.296+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.296+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.296+0000 7fac0c6e4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.296+0000 7fac0c6e4800 20 started sync module instance, tier type = 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.296+0000 7fac0c6e4800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:19.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.297+0000 7fac0a9e5640 0 -- 172.21.15.12:0/311149769 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fabf4030b70 msgr2=0x7fabf403e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=0 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=1 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=2 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=3 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=4 2023-12-03T11:26:19.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.319+0000 7fac0c6e4800 20 add_watcher() i=5 2023-12-03T11:26:19.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.322+0000 7fac0c6e4800 20 add_watcher() i=6 2023-12-03T11:26:19.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.322+0000 7fac0c6e4800 20 add_watcher() i=7 2023-12-03T11:26:19.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.322+0000 7fac0c6e4800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:19.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.324+0000 7fabcffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.324+0000 7fac0c6e4800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:19.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.324+0000 7fac0c6e4800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:19.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fac0c6e4800 5 note: GC not initialized 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fab3effd640 20 reqs_thread_entry: start 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fac0c6e4800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fac0c6e4800 20 Filter name: none 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fab3cff9640 20 reqs_thread_entry: start 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fac0c6e4800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:19.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.330+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 20 rados->read ofs=0 len=0 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:19.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.331+0000 7fac0c6e4800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.332+0000 7fac0c6e4800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.332+0000 7fac0c6e4800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:26:19.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.333+0000 7fabcffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:19.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.335+0000 7fac0c6e4800 20 remove_watcher() i=0 2023-12-03T11:26:19.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.335+0000 7fac0c6e4800 2 removed watcher, disabling cache 2023-12-03T11:26:19.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.336+0000 7fac0c6e4800 20 remove_watcher() i=1 2023-12-03T11:26:19.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.337+0000 7fac0c6e4800 20 remove_watcher() i=2 2023-12-03T11:26:19.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.338+0000 7fac0c6e4800 20 remove_watcher() i=3 2023-12-03T11:26:19.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.340+0000 7fac0c6e4800 20 remove_watcher() i=4 2023-12-03T11:26:19.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.341+0000 7fac0c6e4800 20 remove_watcher() i=5 2023-12-03T11:26:19.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.342+0000 7fac0c6e4800 20 remove_watcher() i=6 2023-12-03T11:26:19.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:19.343+0000 7fac0c6e4800 20 remove_watcher() i=7 2023-12-03T11:26:19.350 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:26:19.744 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:26:21.201 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:21.203 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:26:21.239 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:21.239 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:21.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.255+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 realm 2023-12-03T11:26:21.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:21.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.257+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.258+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.258+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.259+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:21.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.259+0000 7f2bd3521800 20 searching for the correct realm 2023-12-03T11:26:21.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.268+0000 7f2bcfab9640 0 -- 172.21.15.12:0/1679353823 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x56232da1c0f0 msgr2=0x56232da1e4e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.272+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:21.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 zone default found 2023-12-03T11:26:21.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 4 Realm: () 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 zonegroup default 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.273+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.274+0000 7f2bcfab9640 0 -- 172.21.15.12:0/1679353823 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56232da27fb0 msgr2=0x56232da2a470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.274+0000 7f2bd3521800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.274+0000 7f2bd3521800 20 started sync module instance, tier type = 2023-12-03T11:26:21.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.274+0000 7f2bd3521800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:21.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=0 2023-12-03T11:26:21.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=1 2023-12-03T11:26:21.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=2 2023-12-03T11:26:21.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=3 2023-12-03T11:26:21.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=4 2023-12-03T11:26:21.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=5 2023-12-03T11:26:21.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.290+0000 7f2bd3521800 20 add_watcher() i=6 2023-12-03T11:26:21.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.294+0000 7f2bd3521800 20 add_watcher() i=7 2023-12-03T11:26:21.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.294+0000 7f2bd3521800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:21.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.295+0000 7f2b957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.296+0000 7f2bd3521800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:21.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.296+0000 7f2bd3521800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:21.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2bd3521800 5 note: GC not initialized 2023-12-03T11:26:21.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2b03fff640 20 reqs_thread_entry: start 2023-12-03T11:26:21.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2bd3521800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:21.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2bd3521800 20 Filter name: none 2023-12-03T11:26:21.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2b01ffb640 20 reqs_thread_entry: start 2023-12-03T11:26:21.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.302+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.303+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.303+0000 7f2bd3521800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.303+0000 7f2bd3521800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 INFO: main: starting shard=0 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:26:21.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.304+0000 7f2bd3521800 20 INFO: main: done 2023-12-03T11:26:21.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.305+0000 7f2b957e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.307+0000 7f2bd3521800 20 remove_watcher() i=0 2023-12-03T11:26:21.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.307+0000 7f2bd3521800 2 removed watcher, disabling cache 2023-12-03T11:26:21.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.309+0000 7f2bd3521800 20 remove_watcher() i=1 2023-12-03T11:26:21.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.310+0000 7f2bd3521800 20 remove_watcher() i=2 2023-12-03T11:26:21.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.311+0000 7f2bd3521800 20 remove_watcher() i=3 2023-12-03T11:26:21.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.312+0000 7f2bd3521800 20 remove_watcher() i=4 2023-12-03T11:26:21.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.313+0000 7f2bd3521800 20 remove_watcher() i=5 2023-12-03T11:26:21.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.315+0000 7f2bd3521800 20 remove_watcher() i=6 2023-12-03T11:26:21.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.316+0000 7f2bd3521800 20 remove_watcher() i=7 2023-12-03T11:26:21.383 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:21.385 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:26:21.415 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:21.415 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:21.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.428+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 realm 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.430+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.431+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.431+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.432+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:21.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.432+0000 7f1c60323800 20 searching for the correct realm 2023-12-03T11:26:21.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.443+0000 7f1c5c744640 0 -- 172.21.15.12:0/1950330361 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556525efb3e0 msgr2=0x556525efd880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:21.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.444+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 zone default found 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 4 Realm: () 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:21.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 zonegroup default 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.445+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.446+0000 7f1c60323800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.446+0000 7f1c60323800 20 started sync module instance, tier type = 2023-12-03T11:26:21.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.446+0000 7f1c60323800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:21.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.455+0000 7f1c5c744640 0 -- 172.21.15.12:0/1950330361 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f1c34077340 msgr2=0x7f1c34079770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=0 2023-12-03T11:26:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=1 2023-12-03T11:26:21.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=2 2023-12-03T11:26:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=3 2023-12-03T11:26:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=4 2023-12-03T11:26:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=5 2023-12-03T11:26:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=6 2023-12-03T11:26:21.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 20 add_watcher() i=7 2023-12-03T11:26:21.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.461+0000 7f1c60323800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:21.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.463+0000 7f1c21fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.464+0000 7f1c60323800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:21.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.464+0000 7f1c60323800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1c60323800 5 note: GC not initialized 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1b98ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1c60323800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1c60323800 20 Filter name: none 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1b8e7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:21.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.470+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:21.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:21.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:21.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:21.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 INFO: main: starting shard=0 2023-12-03T11:26:21.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.471+0000 7f1c60323800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:21.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.472+0000 7f1c60323800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:26:21.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.472+0000 7f1c60323800 20 INFO: main: done 2023-12-03T11:26:21.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.473+0000 7f1c21fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.475+0000 7f1c60323800 20 remove_watcher() i=0 2023-12-03T11:26:21.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.475+0000 7f1c60323800 2 removed watcher, disabling cache 2023-12-03T11:26:21.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.477+0000 7f1c60323800 20 remove_watcher() i=1 2023-12-03T11:26:21.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.478+0000 7f1c60323800 20 remove_watcher() i=2 2023-12-03T11:26:21.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.479+0000 7f1c60323800 20 remove_watcher() i=3 2023-12-03T11:26:21.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.480+0000 7f1c60323800 20 remove_watcher() i=4 2023-12-03T11:26:21.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.481+0000 7f1c60323800 20 remove_watcher() i=5 2023-12-03T11:26:21.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.485+0000 7f1c60323800 20 remove_watcher() i=6 2023-12-03T11:26:21.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.486+0000 7f1c60323800 20 remove_watcher() i=7 2023-12-03T11:26:21.569 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:21.570 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-12-03T11:26:21.600 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:21.600 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:21.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.613+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 realm 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:21.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.615+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.616+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.616+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.617+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:21.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.617+0000 7fb50b138800 20 searching for the correct realm 2023-12-03T11:26:21.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.628+0000 7fb506e06640 0 -- 172.21.15.12:0/2316224194 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x558481e1d7a0 msgr2=0x558481e1fc60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:21.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:21.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:21.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:21.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.630+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 zone default found 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 4 Realm: () 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:21.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 zonegroup default 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.631+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.632+0000 7fb50b138800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.632+0000 7fb50b138800 20 started sync module instance, tier type = 2023-12-03T11:26:21.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.632+0000 7fb50b138800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:21.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.641+0000 7fb507607640 0 -- 172.21.15.12:0/2316224194 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fb4fc03e770 msgr2=0x7fb4fc08e020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=0 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=1 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=2 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=3 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=4 2023-12-03T11:26:21.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=5 2023-12-03T11:26:21.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.649+0000 7fb50b138800 20 add_watcher() i=6 2023-12-03T11:26:21.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.650+0000 7fb50b138800 20 add_watcher() i=7 2023-12-03T11:26:21.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.650+0000 7fb50b138800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:21.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.652+0000 7fb447fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.652+0000 7fb50b138800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:21.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.652+0000 7fb50b138800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:21.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb50b138800 5 note: GC not initialized 2023-12-03T11:26:21.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb436ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:21.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb50b138800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:21.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb50b138800 20 Filter name: none 2023-12-03T11:26:21.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb434ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:21.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.658+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 INFO: main: starting shard=0 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.659+0000 7fb50b138800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.660+0000 7fb50b138800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:26:21.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.660+0000 7fb50b138800 20 INFO: main: done 2023-12-03T11:26:21.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.660+0000 7fb447fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.664+0000 7fb50b138800 20 remove_watcher() i=0 2023-12-03T11:26:21.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.664+0000 7fb50b138800 2 removed watcher, disabling cache 2023-12-03T11:26:21.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.665+0000 7fb50b138800 20 remove_watcher() i=1 2023-12-03T11:26:21.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.668+0000 7fb50b138800 20 remove_watcher() i=2 2023-12-03T11:26:21.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.669+0000 7fb50b138800 20 remove_watcher() i=3 2023-12-03T11:26:21.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.670+0000 7fb50b138800 20 remove_watcher() i=4 2023-12-03T11:26:21.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.672+0000 7fb50b138800 20 remove_watcher() i=5 2023-12-03T11:26:21.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.673+0000 7fb50b138800 20 remove_watcher() i=6 2023-12-03T11:26:21.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.674+0000 7fb50b138800 20 remove_watcher() i=7 2023-12-03T11:26:21.742 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:21.744 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-12-03T11:26:21.775 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:21.775 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:21.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.789+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.790+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.790+0000 7fc2f9b31800 20 realm 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.790+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.791+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.791+0000 7fc2f9b31800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.791+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.791+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.791+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.792+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.792+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.793+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:21.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.793+0000 7fc2f9b31800 20 searching for the correct realm 2023-12-03T11:26:21.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.802+0000 7fc2f583c640 0 -- 172.21.15.12:0/3290154512 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55cd1a8bac20 msgr2=0x55cd1a8bd010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:21.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 zone default found 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 4 Realm: () 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 zonegroup default 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.804+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:21.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 started sync module instance, tier type = 2023-12-03T11:26:21.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.805+0000 7fc2f9b31800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:21.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.815+0000 7fc2f603d640 0 -- 172.21.15.12:0/3290154512 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55cd1a8c2ca0 msgr2=0x55cd1a8c5130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=0 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=1 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=2 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=3 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=4 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.820+0000 7fc2f9b31800 20 add_watcher() i=5 2023-12-03T11:26:21.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.821+0000 7fc2f9b31800 20 add_watcher() i=6 2023-12-03T11:26:21.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.823+0000 7fc2f9b31800 20 add_watcher() i=7 2023-12-03T11:26:21.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.823+0000 7fc2f9b31800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:21.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.825+0000 7fc2bb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.825+0000 7fc2f9b31800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:21.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.825+0000 7fc2f9b31800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.831+0000 7fc2f9b31800 5 note: GC not initialized 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.831+0000 7fc22a7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.831+0000 7fc2f9b31800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 Filter name: none 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc21bfff640 20 reqs_thread_entry: start 2023-12-03T11:26:21.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 rados->read ofs=0 len=0 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 INFO: main: starting shard=0 2023-12-03T11:26:21.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.832+0000 7fc2f9b31800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:21.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.833+0000 7fc2f9b31800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:26:21.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.833+0000 7fc2f9b31800 20 INFO: main: done 2023-12-03T11:26:21.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.834+0000 7fc2bb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:21.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.836+0000 7fc2f9b31800 20 remove_watcher() i=0 2023-12-03T11:26:21.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.836+0000 7fc2f9b31800 2 removed watcher, disabling cache 2023-12-03T11:26:21.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.837+0000 7fc2f9b31800 20 remove_watcher() i=1 2023-12-03T11:26:21.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.838+0000 7fc2f9b31800 20 remove_watcher() i=2 2023-12-03T11:26:21.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.840+0000 7fc2f9b31800 20 remove_watcher() i=3 2023-12-03T11:26:21.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.841+0000 7fc2f9b31800 20 remove_watcher() i=4 2023-12-03T11:26:21.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.842+0000 7fc2f9b31800 20 remove_watcher() i=5 2023-12-03T11:26:21.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.845+0000 7fc2f9b31800 20 remove_watcher() i=6 2023-12-03T11:26:21.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:21.846+0000 7fc2f9b31800 20 remove_watcher() i=7 2023-12-03T11:26:21.858 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:26:22.255 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:26:22.648 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check unlinked finds unlistable entries 2023-12-03T11:26:22.649 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:22.650 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:26:22.682 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:22.682 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:22.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.695+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 20 realm 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.697+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.698+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.698+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.699+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:22.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.699+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.699+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:22.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.699+0000 7fbc675dc800 20 searching for the correct realm 2023-12-03T11:26:22.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.710+0000 7fbc658dd640 0 -- 172.21.15.12:0/3639519605 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5654b14ebf10 msgr2=0x5654b14ee3d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.711+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:22.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:22.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 zone default found 2023-12-03T11:26:22.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 4 Realm: () 2023-12-03T11:26:22.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:22.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 zonegroup default 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.712+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.713+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.713+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.713+0000 7fbc675dc800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.713+0000 7fbc675dc800 20 started sync module instance, tier type = 2023-12-03T11:26:22.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.713+0000 7fbc675dc800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:22.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.723+0000 7fbc658dd640 0 -- 172.21.15.12:0/3639519605 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7fbc40077370 msgr2=0x7fbc400797a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:22.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=0 2023-12-03T11:26:22.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=1 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=2 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=3 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=4 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.728+0000 7fbc675dc800 20 add_watcher() i=5 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.729+0000 7fbc675dc800 20 add_watcher() i=6 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.729+0000 7fbc675dc800 20 add_watcher() i=7 2023-12-03T11:26:22.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.729+0000 7fbc675dc800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:22.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.731+0000 7fbc2afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:22.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.731+0000 7fbc675dc800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:22.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.731+0000 7fbc675dc800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:22.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.737+0000 7fbc675dc800 5 note: GC not initialized 2023-12-03T11:26:22.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbb99ffb640 20 reqs_thread_entry: start 2023-12-03T11:26:22.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:22.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 20 Filter name: none 2023-12-03T11:26:22.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbb8b7fe640 20 reqs_thread_entry: start 2023-12-03T11:26:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:22.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.738+0000 7fbc675dc800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:22.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.739+0000 7fbc675dc800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.741+0000 7fbc675dc800 1 NOTICE: finished shard 0 (4 entries found) 2023-12-03T11:26:22.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.741+0000 7fbc675dc800 1 NOTICE: finished all shards (4 entries found) 2023-12-03T11:26:22.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.742+0000 7fbc2afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:22.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.745+0000 7fbc675dc800 20 remove_watcher() i=0 2023-12-03T11:26:22.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.745+0000 7fbc675dc800 2 removed watcher, disabling cache 2023-12-03T11:26:22.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.746+0000 7fbc675dc800 20 remove_watcher() i=1 2023-12-03T11:26:22.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.748+0000 7fbc675dc800 20 remove_watcher() i=2 2023-12-03T11:26:22.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.749+0000 7fbc675dc800 20 remove_watcher() i=3 2023-12-03T11:26:22.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.750+0000 7fbc675dc800 20 remove_watcher() i=4 2023-12-03T11:26:22.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.751+0000 7fbc675dc800 20 remove_watcher() i=5 2023-12-03T11:26:22.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.752+0000 7fbc675dc800 20 remove_watcher() i=6 2023-12-03T11:26:22.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.753+0000 7fbc675dc800 20 remove_watcher() i=7 2023-12-03T11:26:22.759 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: unlinked entries are not listable 2023-12-03T11:26:22.759 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:22.766 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: GET returns 404 for unlinked entry keys that have no other versions 2023-12-03T11:26:22.766 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:22.854 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check unlinked fixes unlistable entries 2023-12-03T11:26:22.857 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:22.858 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:26:22.890 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:22.890 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:22.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.903+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 realm 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.905+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.906+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.906+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:22.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.907+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.907+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:22.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.908+0000 7f6e188c0800 20 searching for the correct realm 2023-12-03T11:26:22.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.918+0000 7f6e16bc1640 0 -- 172.21.15.12:0/3307301886 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x555fe7f85ed0 msgr2=0x555fe7f88390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.919+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 zone default found 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 4 Realm: () 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 zonegroup default 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.920+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.921+0000 7f6e188c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:22.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.921+0000 7f6e188c0800 20 started sync module instance, tier type = 2023-12-03T11:26:22.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.921+0000 7f6e188c0800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:22.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.936+0000 7f6e16bc1640 0 -- 172.21.15.12:0/3307301886 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f6e00058950 msgr2=0x7f6e0008c0b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=0 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=1 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=2 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=3 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=4 2023-12-03T11:26:22.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.937+0000 7f6e188c0800 20 add_watcher() i=5 2023-12-03T11:26:22.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.940+0000 7f6e188c0800 20 add_watcher() i=6 2023-12-03T11:26:22.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.942+0000 7f6e188c0800 20 add_watcher() i=7 2023-12-03T11:26:22.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.942+0000 7f6e188c0800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:22.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.945+0000 7f6ddbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:22.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.945+0000 7f6e188c0800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:22.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.945+0000 7f6e188c0800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.951+0000 7f6e188c0800 5 note: GC not initialized 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6d4affd640 20 reqs_thread_entry: start 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 20 Filter name: none 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6d48ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:22.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:22.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.952+0000 7f6e188c0800 20 rados->read ofs=0 len=0 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.953+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.954+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.954+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:c state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:22.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.955+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to 9xrqpp0f87c72d435temrs8t6i7icg3a 2023-12-03T11:26:22.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.955+0000 7f6e188c0800 20 get_obj_state_impl: setting s->olh_tag to kwmsmkbe0ge33qlp9md02v2r6pu6krw2 2023-12-03T11:26:22.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.955+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=kwmsmkbe0ge33qlp9md02v2r6pu6krw2 2023-12-03T11:26:22.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.956+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.959+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 olh_log_entry: epoch=3 op=3 key=c[j0SKG0Xb8LEou1miKelmAd0MFV3c3DP] 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:j0SKG0Xb8LEou1miKelmAd0MFV3c3DP_c state=0x555fe803e2a8 s->prefetch_data=0 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 10 manifest: total_size = 22 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10075242033415530961 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:j0SKG0Xb8LEou1miKelmAd0MFV3c3DP_c state=0x555fe803e2a8 s->prefetch_data=0 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 prepare_atomic_modification: state is not atomic. state=0x555fe803e2a8 2023-12-03T11:26:22.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.960+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.966+0000 7f6e188c0800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:22.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.966+0000 7f6e188c0800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:26:22.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.966+0000 7f6e188c0800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:22.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.966+0000 7f6e188c0800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:j0SKG0Xb8LEou1miKelmAd0MFV3c3DP_.z8N1wgjfZO1GgRakHyUOjITh6CsjKlw_0 2023-12-03T11:26:22.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.972+0000 7f6e0cff9640 20 handle_completion(): completion ok for obj=c[j0SKG0Xb8LEou1miKelmAd0MFV3c3DP] 2023-12-03T11:26:22.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.972+0000 7f6e188c0800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:22.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.975+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.975+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.979 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.978+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.979+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.979+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:d state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.979+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to nkphd3wo2yovh847hgc6ok2u6tt52kzf 2023-12-03T11:26:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.979+0000 7f6e188c0800 20 get_obj_state_impl: setting s->olh_tag to rojyloob44693odo3jo7wtqm8hjvc0rt 2023-12-03T11:26:22.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.979+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=rojyloob44693odo3jo7wtqm8hjvc0rt 2023-12-03T11:26:22.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.981+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.982+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 olh_log_entry: epoch=3 op=3 key=d[aik3ky7wEJKP8Rsq51qJfiRb1vYjjzh] 2023-12-03T11:26:22.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:aik3ky7wEJKP8Rsq51qJfiRb1vYjjzh_d state=0x555fe803d298 s->prefetch_data=0 2023-12-03T11:26:22.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 10 manifest: total_size = 22 2023-12-03T11:26:22.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12907169499789013197 2023-12-03T11:26:22.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:aik3ky7wEJKP8Rsq51qJfiRb1vYjjzh_d state=0x555fe803d298 s->prefetch_data=0 2023-12-03T11:26:22.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 prepare_atomic_modification: state is not atomic. state=0x555fe803d298 2023-12-03T11:26:22.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.983+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.989+0000 7f6e188c0800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:22.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.989+0000 7f6e188c0800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:26:22.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.989+0000 7f6e188c0800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:22.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.989+0000 7f6e188c0800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:aik3ky7wEJKP8Rsq51qJfiRb1vYjjzh_.hRaoQTcMn7-S3tYopLc1I2n1xShEjX-_0 2023-12-03T11:26:22.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.993+0000 7f6df7fff640 20 handle_completion(): completion ok for obj=d[aik3ky7wEJKP8Rsq51qJfiRb1vYjjzh] 2023-12-03T11:26:22.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.993+0000 7f6e188c0800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:22.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.995+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.995+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.997+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.997+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:22.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.997+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:e state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.998+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to 2sxu9rst7qrd5p1m78cj0sjkdgvrno18 2023-12-03T11:26:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.998+0000 7f6e188c0800 20 get_obj_state_impl: setting s->olh_tag to w9c4292h6lbzjsy9um7dth1a8wmgbugh 2023-12-03T11:26:22.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:22.998+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=w9c4292h6lbzjsy9um7dth1a8wmgbugh 2023-12-03T11:26:23.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.000+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.002 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-12-03T11:26:23.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 olh_log_entry: epoch=2 op=3 key=e[ZEspoX5HQ0yMpdyLoKR2.8TW-DeUgrg] 2023-12-03T11:26:23.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:ZEspoX5HQ0yMpdyLoKR2.8TW-DeUgrg_e state=0x555fe803d238 s->prefetch_data=0 2023-12-03T11:26:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 10 manifest: total_size = 9 2023-12-03T11:26:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14428920627874851012 2023-12-03T11:26:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:ZEspoX5HQ0yMpdyLoKR2.8TW-DeUgrg_e state=0x555fe803d238 s->prefetch_data=0 2023-12-03T11:26:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 prepare_atomic_modification: state is not atomic. state=0x555fe803d238 2023-12-03T11:26:23.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.002+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.007+0000 7f6e188c0800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:23.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.007+0000 7f6e188c0800 20 RGWObjManifest::operator++(): result: ofs=9 stripe_ofs=9 part_ofs=0 rule->part_size=0 2023-12-03T11:26:23.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.007+0000 7f6e188c0800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:23.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.007+0000 7f6e188c0800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:ZEspoX5HQ0yMpdyLoKR2.8TW-DeUgrg_.Men2KS1o1Z8iZPBYYlroAgqLQWMcYab_0 2023-12-03T11:26:23.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.012+0000 7f6e0cff9640 20 handle_completion(): completion ok for obj=e[ZEspoX5HQ0yMpdyLoKR2.8TW-DeUgrg] 2023-12-03T11:26:23.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.012+0000 7f6e188c0800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:23.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.014+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:e state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:23.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.014+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=w9c4292h6lbzjsy9um7dth1a8wmgbugh 2023-12-03T11:26:23.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.017+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.020+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.020+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.020+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:f state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:23.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.021+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to zilfwppug666agsgrb2m1efnousrki5g 2023-12-03T11:26:23.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.021+0000 7f6e188c0800 20 get_obj_state_impl: setting s->olh_tag to nnu0vif634xfyb2og63etv4yd5udhf9x 2023-12-03T11:26:23.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.021+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=nnu0vif634xfyb2og63etv4yd5udhf9x 2023-12-03T11:26:23.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.022+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.027+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 olh_log_entry: epoch=2 op=3 key=f[puMTVtOhqm-jGmpQUG-Kj6c0bK4UIK9] 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:puMTVtOhqm-jGmpQUG-Kj6c0bK4UIK9_f state=0x555fe8040ff8 s->prefetch_data=0 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 10 manifest: total_size = 11 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4990281537120188749 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:_:puMTVtOhqm-jGmpQUG-Kj6c0bK4UIK9_f state=0x555fe8040ff8 s->prefetch_data=0 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 prepare_atomic_modification: state is not atomic. state=0x555fe8040ff8 2023-12-03T11:26:23.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.028+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.031+0000 7f6e188c0800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:23.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.031+0000 7f6e188c0800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:26:23.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.031+0000 7f6e188c0800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:23.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.031+0000 7f6e188c0800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:puMTVtOhqm-jGmpQUG-Kj6c0bK4UIK9_.UpnHqJ-amfh57QBzTFNXFoeeRVXZ36w_0 2023-12-03T11:26:23.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.035+0000 7f6df7fff640 20 handle_completion(): completion ok for obj=f[puMTVtOhqm-jGmpQUG-Kj6c0bK4UIK9] 2023-12-03T11:26:23.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.035+0000 7f6e188c0800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:23.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.036+0000 7f6e188c0800 20 get_obj_state: octx=0x7f6e0000a360 obj=check-bucket:f state=0x555fe8034ce8 s->prefetch_data=0 2023-12-03T11:26:23.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.036+0000 7f6e188c0800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=nnu0vif634xfyb2og63etv4yd5udhf9x 2023-12-03T11:26:23.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.046+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.051+0000 7f6e188c0800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.051+0000 7f6e188c0800 1 NOTICE: finished shard 0 (4 entries removed) 2023-12-03T11:26:23.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.051+0000 7f6e188c0800 1 NOTICE: finished all shards (4 entries removed) 2023-12-03T11:26:23.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.052+0000 7f6ddbfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.054+0000 7f6e188c0800 20 remove_watcher() i=0 2023-12-03T11:26:23.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.054+0000 7f6e188c0800 2 removed watcher, disabling cache 2023-12-03T11:26:23.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.056+0000 7f6e188c0800 20 remove_watcher() i=1 2023-12-03T11:26:23.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.057+0000 7f6e188c0800 20 remove_watcher() i=2 2023-12-03T11:26:23.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.058+0000 7f6e188c0800 20 remove_watcher() i=3 2023-12-03T11:26:23.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.059+0000 7f6e188c0800 20 remove_watcher() i=4 2023-12-03T11:26:23.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.060+0000 7f6e188c0800 20 remove_watcher() i=5 2023-12-03T11:26:23.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.061+0000 7f6e188c0800 20 remove_watcher() i=6 2023-12-03T11:26:23.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.062+0000 7f6e188c0800 20 remove_watcher() i=7 2023-12-03T11:26:23.079 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check unlinked does not affect normal entries 2023-12-03T11:26:23.079 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:23.134 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check unlinked does not find new unlistable entries after fix 2023-12-03T11:26:23.134 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:23.135 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --min-age-hours 0 --dump-keys 2023-12-03T11:26:23.167 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.167 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.180+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 realm 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.182+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.183+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.183+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.184+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.184+0000 7f634ba8c800 20 searching for the correct realm 2023-12-03T11:26:23.195 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f6349d8d640 0 -- 172.21.15.12:0/2247748384 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5580e5837f70 msgr2=0x5580e583a430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f634ba8c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f634ba8c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f634ba8c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f634ba8c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.194+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 zone default found 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 4 Realm: () 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 zonegroup default 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.195+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 started sync module instance, tier type = 2023-12-03T11:26:23.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.196+0000 7f634ba8c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.208+0000 7f6349d8d640 0 -- 172.21.15.12:0/2247748384 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f63240772f0 msgr2=0x7f6324079720 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=0 2023-12-03T11:26:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=1 2023-12-03T11:26:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=2 2023-12-03T11:26:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=3 2023-12-03T11:26:23.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=4 2023-12-03T11:26:23.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=5 2023-12-03T11:26:23.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.210+0000 7f634ba8c800 20 add_watcher() i=6 2023-12-03T11:26:23.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.211+0000 7f634ba8c800 20 add_watcher() i=7 2023-12-03T11:26:23.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.211+0000 7f634ba8c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.213+0000 7f630efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.213+0000 7f634ba8c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.213+0000 7f634ba8c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f634ba8c800 5 note: GC not initialized 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f627dffb640 20 reqs_thread_entry: start 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f634ba8c800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f634ba8c800 20 Filter name: none 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f626f7fe640 20 reqs_thread_entry: start 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f634ba8c800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:23.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.219+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:23.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:23.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:23.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.220+0000 7f634ba8c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.221+0000 7f634ba8c800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.221+0000 7f634ba8c800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:26:23.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.222+0000 7f630efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.225+0000 7f634ba8c800 20 remove_watcher() i=0 2023-12-03T11:26:23.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.225+0000 7f634ba8c800 2 removed watcher, disabling cache 2023-12-03T11:26:23.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.228+0000 7f634ba8c800 20 remove_watcher() i=1 2023-12-03T11:26:23.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.229+0000 7f634ba8c800 20 remove_watcher() i=2 2023-12-03T11:26:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.231+0000 7f634ba8c800 20 remove_watcher() i=3 2023-12-03T11:26:23.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.232+0000 7f634ba8c800 20 remove_watcher() i=4 2023-12-03T11:26:23.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.233+0000 7f634ba8c800 20 remove_watcher() i=5 2023-12-03T11:26:23.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.234+0000 7f634ba8c800 20 remove_watcher() i=6 2023-12-03T11:26:23.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.235+0000 7f634ba8c800 20 remove_watcher() i=7 2023-12-03T11:26:23.293 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:26:23.322 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.322 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.335+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.336+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.336+0000 7f65423af800 20 realm 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.336+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.337+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.337+0000 7f65423af800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.337+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.337+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.337+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.338+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.338+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.338+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.338+0000 7f65423af800 20 searching for the correct realm 2023-12-03T11:26:23.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.349+0000 7f65406b0640 0 -- 172.21.15.12:0/310392942 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56275ce8e410 msgr2=0x56275ce908d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.350+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 zone default found 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 4 Realm: () 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 zonegroup default 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 started sync module instance, tier type = 2023-12-03T11:26:23.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.351+0000 7f65423af800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.359+0000 7f65406b0640 0 -- 172.21.15.12:0/310392942 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f6518077340 msgr2=0x7f6518079770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=0 2023-12-03T11:26:23.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=1 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=2 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=3 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=4 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=5 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.363+0000 7f65423af800 20 add_watcher() i=6 2023-12-03T11:26:23.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.364+0000 7f65423af800 20 add_watcher() i=7 2023-12-03T11:26:23.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.364+0000 7f65423af800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.366+0000 7f6505fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.366+0000 7f65423af800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.366+0000 7f65423af800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.372+0000 7f65423af800 5 note: GC not initialized 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.372+0000 7f647cff9640 20 reqs_thread_entry: start 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.372+0000 7f65423af800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 Filter name: none 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f64727fc640 20 reqs_thread_entry: start 2023-12-03T11:26:23.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 INFO: main: starting shard=0 2023-12-03T11:26:23.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.373+0000 7f65423af800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.374+0000 7f65423af800 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2023-12-03T11:26:23.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.374+0000 7f65423af800 20 INFO: main: done 2023-12-03T11:26:23.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.375+0000 7f6505fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.378+0000 7f65423af800 20 remove_watcher() i=0 2023-12-03T11:26:23.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.378+0000 7f65423af800 2 removed watcher, disabling cache 2023-12-03T11:26:23.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.379+0000 7f65423af800 20 remove_watcher() i=1 2023-12-03T11:26:23.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.380+0000 7f65423af800 20 remove_watcher() i=2 2023-12-03T11:26:23.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.381+0000 7f65423af800 20 remove_watcher() i=3 2023-12-03T11:26:23.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.383+0000 7f65423af800 20 remove_watcher() i=4 2023-12-03T11:26:23.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.384+0000 7f65423af800 20 remove_watcher() i=5 2023-12-03T11:26:23.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.386+0000 7f65423af800 20 remove_watcher() i=6 2023-12-03T11:26:23.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.387+0000 7f65423af800 20 remove_watcher() i=7 2023-12-03T11:26:23.394 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:26:23.424 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.424 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.436+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 realm 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.438+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.439+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.439+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.440+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.440+0000 7f4f0bd2f800 20 searching for the correct realm 2023-12-03T11:26:23.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.449+0000 7f4f0797e640 0 -- 172.21.15.12:0/1231740053 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f4efc0a0320 msgr2=0x7f4efc098570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 zone default found 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 4 Realm: () 2023-12-03T11:26:23.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 zonegroup default 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.451+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 started sync module instance, tier type = 2023-12-03T11:26:23.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.452+0000 7f4f0bd2f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.459+0000 7f4f0797e640 0 -- 172.21.15.12:0/1231740053 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f4ee0077340 msgr2=0x7f4ee0079770 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=0 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=1 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=2 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=3 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=4 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=5 2023-12-03T11:26:23.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.469+0000 7f4f0bd2f800 20 add_watcher() i=6 2023-12-03T11:26:23.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.470+0000 7f4f0bd2f800 20 add_watcher() i=7 2023-12-03T11:26:23.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.470+0000 7f4f0bd2f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.472+0000 7f4ecdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.472+0000 7f4f0bd2f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.472+0000 7f4f0bd2f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.478+0000 7f4f0bd2f800 5 note: GC not initialized 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.478+0000 7f4e44ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.478+0000 7f4f0bd2f800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.478+0000 7f4f0bd2f800 20 Filter name: none 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4e3a7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:23.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 INFO: main: starting shard=0 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.479+0000 7f4f0bd2f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.480+0000 7f4f0bd2f800 20 INFO: main: bi_list() returned without error; entries.size()=2, is_truncated=0, marker= 2023-12-03T11:26:23.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.480+0000 7f4f0bd2f800 20 INFO: main: done 2023-12-03T11:26:23.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.481+0000 7f4ecdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.483+0000 7f4f0bd2f800 20 remove_watcher() i=0 2023-12-03T11:26:23.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.483+0000 7f4f0bd2f800 2 removed watcher, disabling cache 2023-12-03T11:26:23.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.484+0000 7f4f0bd2f800 20 remove_watcher() i=1 2023-12-03T11:26:23.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.486+0000 7f4f0bd2f800 20 remove_watcher() i=2 2023-12-03T11:26:23.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.487+0000 7f4f0bd2f800 20 remove_watcher() i=3 2023-12-03T11:26:23.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.488+0000 7f4f0bd2f800 20 remove_watcher() i=4 2023-12-03T11:26:23.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.489+0000 7f4f0bd2f800 20 remove_watcher() i=5 2023-12-03T11:26:23.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.491+0000 7f4f0bd2f800 20 remove_watcher() i=6 2023-12-03T11:26:23.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.492+0000 7f4f0bd2f800 20 remove_watcher() i=7 2023-12-03T11:26:23.498 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check olh finds leftover OLH entries 2023-12-03T11:26:23.498 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:23.499 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-12-03T11:26:23.529 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.529 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.542+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 realm 2023-12-03T11:26:23.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.544+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.545+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.545+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.546+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.546+0000 7f74f492c800 20 searching for the correct realm 2023-12-03T11:26:23.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.554+0000 7f74f06c3640 0 -- 172.21.15.12:0/1487050020 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f74e405e330 msgr2=0x7f74e406b710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.556+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 zone default found 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 4 Realm: () 2023-12-03T11:26:23.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 zonegroup default 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.557+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.558+0000 7f74f492c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.558+0000 7f74f492c800 20 started sync module instance, tier type = 2023-12-03T11:26:23.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.558+0000 7f74f492c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.563+0000 7f74f06c3640 0 -- 172.21.15.12:0/1487050020 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f74e4020060 msgr2=0x7f74e4089950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=0 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=1 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=2 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=3 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=4 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=5 2023-12-03T11:26:23.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.573+0000 7f74f492c800 20 add_watcher() i=6 2023-12-03T11:26:23.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.574+0000 7f74f492c800 20 add_watcher() i=7 2023-12-03T11:26:23.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.574+0000 7f74f492c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.576+0000 7f74b67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.576+0000 7f74f492c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.576+0000 7f74f492c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.582+0000 7f74f492c800 5 note: GC not initialized 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.582+0000 7f74257fa640 20 reqs_thread_entry: start 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.582+0000 7f74f492c800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 20 Filter name: none 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f741effd640 20 reqs_thread_entry: start 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:23.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.583+0000 7f74f492c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.584+0000 7f74f492c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.584+0000 7f74f492c800 1 NOTICE: finished shard 0 (2 entries found) 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.584+0000 7f74f492c800 1 NOTICE: finished all shards (2 entries found) 2023-12-03T11:26:23.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.585+0000 7f74b67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.587+0000 7f74f492c800 20 remove_watcher() i=0 2023-12-03T11:26:23.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.587+0000 7f74f492c800 2 removed watcher, disabling cache 2023-12-03T11:26:23.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.588+0000 7f74f492c800 20 remove_watcher() i=1 2023-12-03T11:26:23.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.589+0000 7f74f492c800 20 remove_watcher() i=2 2023-12-03T11:26:23.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.590+0000 7f74f492c800 20 remove_watcher() i=3 2023-12-03T11:26:23.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.591+0000 7f74f492c800 20 remove_watcher() i=4 2023-12-03T11:26:23.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.592+0000 7f74f492c800 20 remove_watcher() i=5 2023-12-03T11:26:23.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.594+0000 7f74f492c800 20 remove_watcher() i=6 2023-12-03T11:26:23.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.597+0000 7f74f492c800 20 remove_watcher() i=7 2023-12-03T11:26:23.603 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check olh fixes leftover OLH entries 2023-12-03T11:26:23.603 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:23.604 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check olh --bucket check-bucket --fix --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:26:23.636 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.636 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.649+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 realm 2023-12-03T11:26:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.651+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.652+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.652+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.653+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.653+0000 7fce8bf25800 20 searching for the correct realm 2023-12-03T11:26:23.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.662+0000 7fce87b54640 0 -- 172.21.15.12:0/2190376499 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55e920c3fcf0 msgr2=0x55e920c42140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 zone default found 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 4 Realm: () 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 zonegroup default 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.664+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.665+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.665+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.665+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.665+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.666+0000 7fce8bf25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.666+0000 7fce8bf25800 20 started sync module instance, tier type = 2023-12-03T11:26:23.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.666+0000 7fce8bf25800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.672+0000 7fce88355640 0 -- 172.21.15.12:0/2190376499 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55e920c47e20 msgr2=0x55e920c4a2e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=0 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=1 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=2 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=3 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=4 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=5 2023-12-03T11:26:23.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.679+0000 7fce8bf25800 20 add_watcher() i=6 2023-12-03T11:26:23.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.680+0000 7fce8bf25800 20 add_watcher() i=7 2023-12-03T11:26:23.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.680+0000 7fce8bf25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.683+0000 7fce4dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.683+0000 7fce8bf25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.683+0000 7fce8bf25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.689+0000 7fce8bf25800 5 note: GC not initialized 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.689+0000 7fcdc4ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 20 Filter name: none 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fcdba7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:23.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:23.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.690+0000 7fce8bf25800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.691+0000 7fce8bf25800 20 get_obj_state: octx=0x7fce8000dfa0 obj=check-bucket:c state=0x55e920cf30a8 s->prefetch_data=0 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 get_obj_state: setting s->obj_tag to 9xrqpp0f87c72d435temrs8t6i7icg3a 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 get_obj_state_impl: setting s->olh_tag to kwmsmkbe0ge33qlp9md02v2r6pu6krw2 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 olh_log_entry: epoch=4 op=2 key=c[] 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 olh_log_entry: epoch=4 op=3 key=c[.OW8BVHHncsez0T-1SThAvGbp3B22ew] 2023-12-03T11:26:23.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.692+0000 7fce8bf25800 20 get_obj_state: octx=0x55e920cf24f0 obj=check-bucket:_:.OW8BVHHncsez0T-1SThAvGbp3B22ew_c state=0x55e920cfcba8 s->prefetch_data=0 2023-12-03T11:26:23.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.694+0000 7fce8bf25800 20 get_obj_state: octx=0x55e920cf24f0 obj=check-bucket:c state=0x55e920cf5b38 s->prefetch_data=0 2023-12-03T11:26:23.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.694+0000 7fce8bf25800 20 get_obj_state: setting s->obj_tag to 9xrqpp0f87c72d435temrs8t6i7icg3a 2023-12-03T11:26:23.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.694+0000 7fce8bf25800 20 get_obj_state_impl: setting s->olh_tag to kwmsmkbe0ge33qlp9md02v2r6pu6krw2 2023-12-03T11:26:23.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.694+0000 7fce8bf25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=kwmsmkbe0ge33qlp9md02v2r6pu6krw2 2023-12-03T11:26:23.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.705+0000 7fce8bf25800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 get_obj_state: octx=0x7fce8000dfa0 obj=check-bucket:d state=0x55e920cf30a8 s->prefetch_data=0 2023-12-03T11:26:23.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 get_obj_state: setting s->obj_tag to nkphd3wo2yovh847hgc6ok2u6tt52kzf 2023-12-03T11:26:23.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 get_obj_state_impl: setting s->olh_tag to rojyloob44693odo3jo7wtqm8hjvc0rt 2023-12-03T11:26:23.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 olh_log_entry: epoch=4 op=2 key=d[] 2023-12-03T11:26:23.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 olh_log_entry: epoch=4 op=3 key=d[kmDPE4V5Y4f4OLQrFEdv8RGvS8EHEIo] 2023-12-03T11:26:23.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.710+0000 7fce8bf25800 20 get_obj_state: octx=0x55e920cf24f0 obj=check-bucket:_:kmDPE4V5Y4f4OLQrFEdv8RGvS8EHEIo_d state=0x55e920cfdbb8 s->prefetch_data=0 2023-12-03T11:26:23.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.712+0000 7fce8bf25800 20 get_obj_state: octx=0x55e920cf24f0 obj=check-bucket:d state=0x55e920cf8b68 s->prefetch_data=0 2023-12-03T11:26:23.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.713+0000 7fce8bf25800 20 get_obj_state: setting s->obj_tag to nkphd3wo2yovh847hgc6ok2u6tt52kzf 2023-12-03T11:26:23.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.713+0000 7fce8bf25800 20 get_obj_state_impl: setting s->olh_tag to rojyloob44693odo3jo7wtqm8hjvc0rt 2023-12-03T11:26:23.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.713+0000 7fce8bf25800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=rojyloob44693odo3jo7wtqm8hjvc0rt 2023-12-03T11:26:23.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.719+0000 7fce8bf25800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.722+0000 7fce8bf25800 1 NOTICE: finished shard 0 (2 entries removed) 2023-12-03T11:26:23.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.722+0000 7fce8bf25800 1 NOTICE: finished all shards (2 entries removed) 2023-12-03T11:26:23.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.723+0000 7fce4dfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.725+0000 7fce8bf25800 20 remove_watcher() i=0 2023-12-03T11:26:23.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.725+0000 7fce8bf25800 2 removed watcher, disabling cache 2023-12-03T11:26:23.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.727+0000 7fce8bf25800 20 remove_watcher() i=1 2023-12-03T11:26:23.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.728+0000 7fce8bf25800 20 remove_watcher() i=2 2023-12-03T11:26:23.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.729+0000 7fce8bf25800 20 remove_watcher() i=3 2023-12-03T11:26:23.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.730+0000 7fce8bf25800 20 remove_watcher() i=4 2023-12-03T11:26:23.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.731+0000 7fce8bf25800 20 remove_watcher() i=5 2023-12-03T11:26:23.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.732+0000 7fce8bf25800 20 remove_watcher() i=6 2023-12-03T11:26:23.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.733+0000 7fce8bf25800 20 remove_watcher() i=7 2023-12-03T11:26:23.741 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:26:23.770 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.770 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.784+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.785+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.785+0000 7f0c3c533800 20 realm 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.785+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.786+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.786+0000 7f0c3c533800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.786+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.786+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.786+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.787+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.787+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.787+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.787+0000 7f0c3c533800 20 searching for the correct realm 2023-12-03T11:26:23.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.796+0000 7f0c381d7640 0 -- 172.21.15.12:0/2713904627 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f0c2c06c040 msgr2=0x7f0c2c06e950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.798+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 zone default found 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 4 Realm: () 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 zonegroup default 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.799+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.800+0000 7f0c3c533800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.800+0000 7f0c3c533800 20 started sync module instance, tier type = 2023-12-03T11:26:23.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.800+0000 7f0c3c533800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.807+0000 7f0c389d8640 0 -- 172.21.15.12:0/2713904627 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f0c2c07f440 msgr2=0x7f0c2c07f8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=0 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=1 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=2 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=3 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=4 2023-12-03T11:26:23.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=5 2023-12-03T11:26:23.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.811+0000 7f0c3c533800 20 add_watcher() i=6 2023-12-03T11:26:23.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.812+0000 7f0c3c533800 20 add_watcher() i=7 2023-12-03T11:26:23.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.812+0000 7f0c3c533800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.813+0000 7f0bfe7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.814+0000 7f0c3c533800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.814+0000 7f0c3c533800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0c3c533800 5 note: GC not initialized 2023-12-03T11:26:23.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0b6d7fa640 20 reqs_thread_entry: start 2023-12-03T11:26:23.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0c3c533800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0c3c533800 20 Filter name: none 2023-12-03T11:26:23.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0b66ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:23.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.820+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:23.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:23.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 INFO: main: starting shard=0 2023-12-03T11:26:23.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.821+0000 7f0c3c533800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.822+0000 7f0c3c533800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:26:23.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.822+0000 7f0c3c533800 20 INFO: main: done 2023-12-03T11:26:23.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.823+0000 7f0bfe7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.825+0000 7f0c3c533800 20 remove_watcher() i=0 2023-12-03T11:26:23.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.825+0000 7f0c3c533800 2 removed watcher, disabling cache 2023-12-03T11:26:23.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.826+0000 7f0c3c533800 20 remove_watcher() i=1 2023-12-03T11:26:23.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.827+0000 7f0c3c533800 20 remove_watcher() i=2 2023-12-03T11:26:23.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.828+0000 7f0c3c533800 20 remove_watcher() i=3 2023-12-03T11:26:23.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.829+0000 7f0c3c533800 20 remove_watcher() i=4 2023-12-03T11:26:23.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.830+0000 7f0c3c533800 20 remove_watcher() i=5 2023-12-03T11:26:23.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.831+0000 7f0c3c533800 20 remove_watcher() i=6 2023-12-03T11:26:23.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.832+0000 7f0c3c533800 20 remove_watcher() i=7 2023-12-03T11:26:23.840 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:26:23.870 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.871 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.883+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 realm 2023-12-03T11:26:23.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.885+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.886+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.886+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.887+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.887+0000 7f71a6535800 20 searching for the correct realm 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a2ace640 0 -- 172.21.15.12:0/1067836316 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x55bbc4aaea10 msgr2=0x55bbc4ab0e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.898+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 zone default found 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 4 Realm: () 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 zonegroup default 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 started sync module instance, tier type = 2023-12-03T11:26:23.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.899+0000 7f71a6535800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=0 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=1 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=2 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=3 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=4 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=5 2023-12-03T11:26:23.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a6535800 20 add_watcher() i=6 2023-12-03T11:26:23.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.911+0000 7f71a22cd640 0 -- 172.21.15.12:0/1067836316 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f71980a0870 msgr2=0x7f7198054040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:23.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.912+0000 7f71a6535800 20 add_watcher() i=7 2023-12-03T11:26:23.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.912+0000 7f71a6535800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:23.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.914+0000 7f7167fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.914+0000 7f71a6535800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:23.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.914+0000 7f71a6535800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.920+0000 7f71a6535800 5 note: GC not initialized 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.920+0000 7f70d6ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f71a6535800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f71a6535800 20 Filter name: none 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f70d4ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:23.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.921+0000 7f71a6535800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 INFO: main: starting shard=0 2023-12-03T11:26:23.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:23.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:26:23.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.922+0000 7f71a6535800 20 INFO: main: done 2023-12-03T11:26:23.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.923+0000 7f7167fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:23.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.925+0000 7f71a6535800 20 remove_watcher() i=0 2023-12-03T11:26:23.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.925+0000 7f71a6535800 2 removed watcher, disabling cache 2023-12-03T11:26:23.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.926+0000 7f71a6535800 20 remove_watcher() i=1 2023-12-03T11:26:23.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.927+0000 7f71a6535800 20 remove_watcher() i=2 2023-12-03T11:26:23.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.928+0000 7f71a6535800 20 remove_watcher() i=3 2023-12-03T11:26:23.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.930+0000 7f71a6535800 20 remove_watcher() i=4 2023-12-03T11:26:23.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.931+0000 7f71a6535800 20 remove_watcher() i=5 2023-12-03T11:26:23.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.932+0000 7f71a6535800 20 remove_watcher() i=6 2023-12-03T11:26:23.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.933+0000 7f71a6535800 20 remove_watcher() i=7 2023-12-03T11:26:23.939 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check olh does not find new leftover OLH entries after fix 2023-12-03T11:26:23.939 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:23.940 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check olh --bucket check-bucket --dump-keys 2023-12-03T11:26:23.970 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:23.970 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:23.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.983+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 20 realm 2023-12-03T11:26:23.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:23.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.985+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.986+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:23.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.986+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.986+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:23.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.986+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:23.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.987+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:23.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.987+0000 7fd143321800 20 searching for the correct realm 2023-12-03T11:26:23.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.997+0000 7fd13f0b8640 0 -- 172.21.15.12:0/3177128634 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55926420e660 msgr2=0x559264210a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:23.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:23.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:24.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:24.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:24.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:24.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.998+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.002 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 zone default found 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 4 Realm: () 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:24.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 zonegroup default 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:23.999+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.000+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:24.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.000+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.000+0000 7fd143321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:24.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.000+0000 7fd143321800 20 started sync module instance, tier type = 2023-12-03T11:26:24.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.000+0000 7fd143321800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:24.009 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.008+0000 7fd13f8b9640 0 -- 172.21.15.12:0/3177128634 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fd128076c20 msgr2=0x7fd1280860b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=0 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=1 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=2 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=3 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=4 2023-12-03T11:26:24.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=5 2023-12-03T11:26:24.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.018+0000 7fd143321800 20 add_watcher() i=6 2023-12-03T11:26:24.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.020+0000 7fd143321800 20 add_watcher() i=7 2023-12-03T11:26:24.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.020+0000 7fd143321800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:24.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.021+0000 7fd1057e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:24.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.022+0000 7fd143321800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:24.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.022+0000 7fd143321800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd143321800 5 note: GC not initialized 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd073fff640 20 reqs_thread_entry: start 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd143321800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd143321800 20 Filter name: none 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd071ffb640 20 reqs_thread_entry: start 2023-12-03T11:26:24.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd143321800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.028+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:24.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 20 rados->read ofs=0 len=0 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:24.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:24.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:24.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 1 NOTICE: finished shard 0 (0 entries found) 2023-12-03T11:26:24.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.029+0000 7fd143321800 1 NOTICE: finished all shards (0 entries found) 2023-12-03T11:26:24.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.030+0000 7fd1057e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:24.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.032+0000 7fd143321800 20 remove_watcher() i=0 2023-12-03T11:26:24.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.032+0000 7fd143321800 2 removed watcher, disabling cache 2023-12-03T11:26:24.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.034+0000 7fd143321800 20 remove_watcher() i=1 2023-12-03T11:26:24.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.035+0000 7fd143321800 20 remove_watcher() i=2 2023-12-03T11:26:24.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.036+0000 7fd143321800 20 remove_watcher() i=3 2023-12-03T11:26:24.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.037+0000 7fd143321800 20 remove_watcher() i=4 2023-12-03T11:26:24.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.039+0000 7fd143321800 20 remove_watcher() i=5 2023-12-03T11:26:24.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.041+0000 7fd143321800 20 remove_watcher() i=6 2023-12-03T11:26:24.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:24.042+0000 7fd143321800 20 remove_watcher() i=7 2023-12-03T11:26:24.048 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: verify that bucket check fixes do not affect null version objects 2023-12-03T11:26:24.048 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:24.099 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: bucket check stats are correct in the presence of unlinked entries 2023-12-03T11:26:24.099 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:26:24.332 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:26:24.721 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:26:26.177 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:26.178 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object c 2023-12-03T11:26:26.213 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:26.213 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:26.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.228+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 realm 2023-12-03T11:26:26.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:26.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.230+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.231+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.231+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.232+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:26.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.232+0000 7f34cfba9800 20 searching for the correct realm 2023-12-03T11:26:26.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.243+0000 7f34cd6a9640 0 -- 172.21.15.12:0/4233639379 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5628b804c880 msgr2=0x5628b804ec70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.244+0000 7f34cfba9800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.244+0000 7f34cfba9800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.244+0000 7f34cfba9800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.244+0000 7f34cfba9800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.244+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 zone default found 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 4 Realm: () 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 zonegroup default 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.245+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 started sync module instance, tier type = 2023-12-03T11:26:26.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.246+0000 7f34cfba9800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:26.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.259+0000 7f34cdeaa640 0 -- 172.21.15.12:0/4233639379 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5628b80548b0 msgr2=0x5628b8056d70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=0 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=1 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=2 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=3 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=4 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=5 2023-12-03T11:26:26.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.264+0000 7f34cfba9800 20 add_watcher() i=6 2023-12-03T11:26:26.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.266+0000 7f34cfba9800 20 add_watcher() i=7 2023-12-03T11:26:26.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.266+0000 7f34cfba9800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:26.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.268+0000 7f3492fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.268+0000 7f34cfba9800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:26.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.268+0000 7f34cfba9800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f34cfba9800 5 note: GC not initialized 2023-12-03T11:26:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f3401ffb640 20 reqs_thread_entry: start 2023-12-03T11:26:26.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f34cfba9800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f34cfba9800 20 Filter name: none 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f33f37fe640 20 reqs_thread_entry: start 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.275+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 INFO: main: starting shard=0 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.276+0000 7f34cfba9800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.277+0000 7f34cfba9800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:26:26.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.277+0000 7f34cfba9800 20 INFO: main: done 2023-12-03T11:26:26.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.278+0000 7f3492fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.280+0000 7f34cfba9800 20 remove_watcher() i=0 2023-12-03T11:26:26.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.280+0000 7f34cfba9800 2 removed watcher, disabling cache 2023-12-03T11:26:26.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.282+0000 7f34cfba9800 20 remove_watcher() i=1 2023-12-03T11:26:26.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.283+0000 7f34cfba9800 20 remove_watcher() i=2 2023-12-03T11:26:26.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.285+0000 7f34cfba9800 20 remove_watcher() i=3 2023-12-03T11:26:26.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.286+0000 7f34cfba9800 20 remove_watcher() i=4 2023-12-03T11:26:26.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.287+0000 7f34cfba9800 20 remove_watcher() i=5 2023-12-03T11:26:26.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.288+0000 7f34cfba9800 20 remove_watcher() i=6 2023-12-03T11:26:26.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.289+0000 7f34cfba9800 20 remove_watcher() i=7 2023-12-03T11:26:26.362 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:26.364 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object d 2023-12-03T11:26:26.394 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:26.394 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:26.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.408+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 realm 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:26.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.410+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.411+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.411+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.412+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:26.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.412+0000 7ff212485800 20 searching for the correct realm 2023-12-03T11:26:26.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.422+0000 7ff20ff85640 0 -- 172.21.15.12:0/336724415 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x560adad72060 msgr2=0x560adad74450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.423+0000 7ff212485800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.423+0000 7ff212485800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.423+0000 7ff212485800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.423+0000 7ff212485800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.423+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 zone default found 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 4 Realm: () 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 zonegroup default 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.424+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.425+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.425+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.425+0000 7ff212485800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.425+0000 7ff212485800 20 started sync module instance, tier type = 2023-12-03T11:26:26.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.425+0000 7ff212485800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:26.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.432+0000 7ff210786640 0 -- 172.21.15.12:0/336724415 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7ff2040c8940 msgr2=0x7ff2040b29b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=0 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=1 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=2 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=3 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=4 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=5 2023-12-03T11:26:26.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.435+0000 7ff212485800 20 add_watcher() i=6 2023-12-03T11:26:26.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.436+0000 7ff212485800 20 add_watcher() i=7 2023-12-03T11:26:26.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.436+0000 7ff212485800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.437+0000 7ff1d5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.438+0000 7ff212485800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.438+0000 7ff212485800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:26.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff212485800 5 note: GC not initialized 2023-12-03T11:26:26.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff14cff9640 20 reqs_thread_entry: start 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff212485800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff212485800 20 Filter name: none 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff1427fc640 20 reqs_thread_entry: start 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.444+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:26.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 INFO: main: starting shard=0 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.445+0000 7ff212485800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.446+0000 7ff212485800 20 INFO: main: bi_list() returned without error; entries.size()=5, is_truncated=0, marker= 2023-12-03T11:26:26.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.446+0000 7ff212485800 20 INFO: main: done 2023-12-03T11:26:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.446+0000 7ff1d5fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.449+0000 7ff212485800 20 remove_watcher() i=0 2023-12-03T11:26:26.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.449+0000 7ff212485800 2 removed watcher, disabling cache 2023-12-03T11:26:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.450+0000 7ff212485800 20 remove_watcher() i=1 2023-12-03T11:26:26.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.451+0000 7ff212485800 20 remove_watcher() i=2 2023-12-03T11:26:26.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.452+0000 7ff212485800 20 remove_watcher() i=3 2023-12-03T11:26:26.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.453+0000 7ff212485800 20 remove_watcher() i=4 2023-12-03T11:26:26.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.454+0000 7ff212485800 20 remove_watcher() i=5 2023-12-03T11:26:26.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.455+0000 7ff212485800 20 remove_watcher() i=6 2023-12-03T11:26:26.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.456+0000 7ff212485800 20 remove_watcher() i=7 2023-12-03T11:26:26.520 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:26.521 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object e 2023-12-03T11:26:26.551 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:26.551 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:26.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.565+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 realm 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:26.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.567+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.568+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.568+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.569+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:26.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.569+0000 7f2093923800 20 searching for the correct realm 2023-12-03T11:26:26.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.576+0000 7f208ed3b640 0 -- 172.21.15.12:0/306271262 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55e2e885a690 msgr2=0x55e2e885ca80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 zone default found 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 4 Realm: () 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:26.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 zonegroup default 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.580+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.581+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.581+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.581+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.581+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.581+0000 7f208fd3d640 0 -- 172.21.15.12:0/306271262 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55e2e885ef70 msgr2=0x55e2e8861430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.582+0000 7f2093923800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.582+0000 7f2093923800 20 started sync module instance, tier type = 2023-12-03T11:26:26.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.582+0000 7f2093923800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:26.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=0 2023-12-03T11:26:26.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=1 2023-12-03T11:26:26.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=2 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=3 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=4 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=5 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.592+0000 7f2093923800 20 add_watcher() i=6 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.593+0000 7f2093923800 20 add_watcher() i=7 2023-12-03T11:26:26.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.593+0000 7f2093923800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:26.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.595+0000 7f20557e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.596+0000 7f2093923800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:26.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.596+0000 7f2093923800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f2093923800 5 note: GC not initialized 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f1fc3fff640 20 reqs_thread_entry: start 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f2093923800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f2093923800 20 Filter name: none 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f1fc1ffb640 20 reqs_thread_entry: start 2023-12-03T11:26:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.602+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:26.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 INFO: main: starting shard=0 2023-12-03T11:26:26.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.603+0000 7f2093923800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:26.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.604+0000 7f2093923800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:26:26.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.604+0000 7f2093923800 20 INFO: main: done 2023-12-03T11:26:26.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.605+0000 7f20557e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.607+0000 7f2093923800 20 remove_watcher() i=0 2023-12-03T11:26:26.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.607+0000 7f2093923800 2 removed watcher, disabling cache 2023-12-03T11:26:26.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.608+0000 7f2093923800 20 remove_watcher() i=1 2023-12-03T11:26:26.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.609+0000 7f2093923800 20 remove_watcher() i=2 2023-12-03T11:26:26.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.610+0000 7f2093923800 20 remove_watcher() i=3 2023-12-03T11:26:26.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.611+0000 7f2093923800 20 remove_watcher() i=4 2023-12-03T11:26:26.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.613+0000 7f2093923800 20 remove_watcher() i=5 2023-12-03T11:26:26.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.613+0000 7f2093923800 20 remove_watcher() i=6 2023-12-03T11:26:26.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.614+0000 7f2093923800 20 remove_watcher() i=7 2023-12-03T11:26:26.676 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:26:26.678 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket check-bucket --object f 2023-12-03T11:26:26.707 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:26.707 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:26.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.721+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 20 realm 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.722+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.723+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.723+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.723+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.723+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.724+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:26.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.724+0000 7fc11bd37800 20 searching for the correct realm 2023-12-03T11:26:26.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.733+0000 7fc117acd640 0 -- 172.21.15.12:0/4107593811 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5623184851e0 msgr2=0x5623184875f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.735+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 zone default found 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 4 Realm: () 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 zonegroup default 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 started sync module instance, tier type = 2023-12-03T11:26:26.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.736+0000 7fc11bd37800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:26.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.746+0000 7fc1182ce640 0 -- 172.21.15.12:0/4107593811 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56231848d300 msgr2=0x56231848f7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:26.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=0 2023-12-03T11:26:26.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=1 2023-12-03T11:26:26.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=2 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=3 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=4 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=5 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.750+0000 7fc11bd37800 20 add_watcher() i=6 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.751+0000 7fc11bd37800 20 add_watcher() i=7 2023-12-03T11:26:26.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.751+0000 7fc11bd37800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:26.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.754+0000 7fc0ddfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.754+0000 7fc11bd37800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:26.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.754+0000 7fc11bd37800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc11bd37800 5 note: GC not initialized 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc054ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc11bd37800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc11bd37800 20 Filter name: none 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc04a7fc640 20 reqs_thread_entry: start 2023-12-03T11:26:26.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.760+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 rados->read ofs=0 len=0 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 INFO: main: max_entries=1000, index=gen=0, max_shards=1 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 INFO: main: starting shard=0 2023-12-03T11:26:26.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.761+0000 7fc11bd37800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:26.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.762+0000 7fc11bd37800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:26:26.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.762+0000 7fc11bd37800 20 INFO: main: done 2023-12-03T11:26:26.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.762+0000 7fc0ddfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:26.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.765+0000 7fc11bd37800 20 remove_watcher() i=0 2023-12-03T11:26:26.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.765+0000 7fc11bd37800 2 removed watcher, disabling cache 2023-12-03T11:26:26.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.766+0000 7fc11bd37800 20 remove_watcher() i=1 2023-12-03T11:26:26.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.767+0000 7fc11bd37800 20 remove_watcher() i=2 2023-12-03T11:26:26.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.769+0000 7fc11bd37800 20 remove_watcher() i=3 2023-12-03T11:26:26.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.770+0000 7fc11bd37800 20 remove_watcher() i=4 2023-12-03T11:26:26.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.771+0000 7fc11bd37800 20 remove_watcher() i=5 2023-12-03T11:26:26.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.772+0000 7fc11bd37800 20 remove_watcher() i=6 2023-12-03T11:26:26.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:26.773+0000 7fc11bd37800 20 remove_watcher() i=7 2023-12-03T11:26:26.785 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:26:27.185 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:26:27.578 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check --fix --bucket check-bucket 2023-12-03T11:26:27.606 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:27.607 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:27.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.620+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.621+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.621+0000 7f4f6b327800 20 realm 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.621+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.622+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.622+0000 7f4f6b327800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.622+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.622+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.622+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.623+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:27.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.623+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.624+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:27.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.624+0000 7f4f6b327800 20 searching for the correct realm 2023-12-03T11:26:27.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.634+0000 7f4f66ff4640 0 -- 172.21.15.12:0/1416905526 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556e78729ff0 msgr2=0x556e7872c400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.635+0000 7f4f6b327800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.635+0000 7f4f6b327800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.635+0000 7f4f6b327800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.635+0000 7f4f6b327800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.635+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:27.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 zone default found 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 4 Realm: () 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 zonegroup default 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.636+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 started sync module instance, tier type = 2023-12-03T11:26:27.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.637+0000 7f4f6b327800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:27.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.645+0000 7f4f66ff4640 0 -- 172.21.15.12:0/1416905526 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f4f5c057490 msgr2=0x7f4f5c05d040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=0 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=1 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=2 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=3 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=4 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=5 2023-12-03T11:26:27.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.655+0000 7f4f6b327800 20 add_watcher() i=6 2023-12-03T11:26:27.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.656+0000 7f4f6b327800 20 add_watcher() i=7 2023-12-03T11:26:27.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.656+0000 7f4f6b327800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:27.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.657+0000 7f4ea7fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:27.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.658+0000 7f4f6b327800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:27.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.658+0000 7f4f6b327800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4f6b327800 5 note: GC not initialized 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4e96ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4f6b327800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4f6b327800 20 Filter name: none 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4e94ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4f6b327800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.664+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.665+0000 7f4f6b327800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.666+0000 7f4f6b327800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.666+0000 7f4f6b327800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.666+0000 7f4f6b327800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:26:27.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.666+0000 7f4f6b327800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:26:27.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.669+0000 7f4ea7fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:27.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.671+0000 7f4f6b327800 20 remove_watcher() i=0 2023-12-03T11:26:27.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.671+0000 7f4f6b327800 2 removed watcher, disabling cache 2023-12-03T11:26:27.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.673+0000 7f4f6b327800 20 remove_watcher() i=1 2023-12-03T11:26:27.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.674+0000 7f4f6b327800 20 remove_watcher() i=2 2023-12-03T11:26:27.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.675+0000 7f4f6b327800 20 remove_watcher() i=3 2023-12-03T11:26:27.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.676+0000 7f4f6b327800 20 remove_watcher() i=4 2023-12-03T11:26:27.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.678+0000 7f4f6b327800 20 remove_watcher() i=5 2023-12-03T11:26:27.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.679+0000 7f4f6b327800 20 remove_watcher() i=6 2023-12-03T11:26:27.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.680+0000 7f4f6b327800 20 remove_watcher() i=7 2023-12-03T11:26:27.687 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket check-bucket --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:26:27.720 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:27.721 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:27.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.734+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 realm 2023-12-03T11:26:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:27.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.736+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.737+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:27.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.737+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.738+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:27.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.738+0000 7fcb62811800 20 searching for the correct realm 2023-12-03T11:26:27.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.749+0000 7fcb60311640 0 -- 172.21.15.12:0/2864379588 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x562d4ff511c0 msgr2=0x562d4ff535b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.750+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:27.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:27.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 zone default found 2023-12-03T11:26:27.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 4 Realm: () 2023-12-03T11:26:27.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 zonegroup default 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.751+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.752+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.752+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.752+0000 7fcb62811800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.752+0000 7fcb62811800 20 started sync module instance, tier type = 2023-12-03T11:26:27.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.752+0000 7fcb62811800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:27.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.759+0000 7fcb60b12640 0 -- 172.21.15.12:0/2864379588 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x562d4ff59280 msgr2=0x562d4ff5b740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:27.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=0 2023-12-03T11:26:27.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=1 2023-12-03T11:26:27.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=2 2023-12-03T11:26:27.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=3 2023-12-03T11:26:27.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=4 2023-12-03T11:26:27.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.765+0000 7fcb62811800 20 add_watcher() i=5 2023-12-03T11:26:27.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.767+0000 7fcb62811800 20 add_watcher() i=6 2023-12-03T11:26:27.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.767+0000 7fcb62811800 20 add_watcher() i=7 2023-12-03T11:26:27.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.767+0000 7fcb62811800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:27.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.769+0000 7fcb267e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:27.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.769+0000 7fcb62811800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:27.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.769+0000 7fcb62811800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 5 note: GC not initialized 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fca957fa640 20 reqs_thread_entry: start 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 20 Filter name: none 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fca8effd640 20 reqs_thread_entry: start 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 10 cache get: name=default.rgw.meta+root+check-bucket : miss 2023-12-03T11:26:27.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:27.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 10 cache put: name=default.rgw.meta+root+check-bucket info.flags=0x11 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 10 adding default.rgw.meta+root+check-bucket to cache LRU end 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 10 cache get: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 : miss 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.776+0000 7fcb62811800 20 rados->read ofs=0 len=0 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 10 cache put: name=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 info.flags=0x17 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 10 adding default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 to cache LRU end 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 10 updating xattr: name=user.rgw.acl bl.length()=179 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.check-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1 2023-12-03T11:26:27.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.777+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.778+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.778+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:c state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.779+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to d4c3d2zcg81zfc2lb9uio96tgr48s7g3 2023-12-03T11:26:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.779+0000 7fcb62811800 20 get_obj_state_impl: setting s->olh_tag to 84q5g4axinahvv2ue3catcev92ymvrgo 2023-12-03T11:26:27.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.779+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=84q5g4axinahvv2ue3catcev92ymvrgo 2023-12-03T11:26:27.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.780+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.784+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 olh_log_entry: epoch=3 op=3 key=c[IGroskPE9L6dLKDXodWaUHsCNoUknLv] 2023-12-03T11:26:27.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:IGroskPE9L6dLKDXodWaUHsCNoUknLv_c state=0x562d5000d988 s->prefetch_data=0 2023-12-03T11:26:27.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 10 manifest: total_size = 22 2023-12-03T11:26:27.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14664029093448718216 2023-12-03T11:26:27.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:IGroskPE9L6dLKDXodWaUHsCNoUknLv_c state=0x562d5000d988 s->prefetch_data=0 2023-12-03T11:26:27.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 prepare_atomic_modification: state is not atomic. state=0x562d5000d988 2023-12-03T11:26:27.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.785+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.788+0000 7fcb62811800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:27.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.788+0000 7fcb62811800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:26:27.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.788+0000 7fcb62811800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:27.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.788+0000 7fcb62811800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:IGroskPE9L6dLKDXodWaUHsCNoUknLv_.lJXd9OYIgOjyf8K95bQ2_AW1q-apai8_0 2023-12-03T11:26:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.792+0000 7fcb56ffd640 20 handle_completion(): completion ok for obj=c[IGroskPE9L6dLKDXodWaUHsCNoUknLv] 2023-12-03T11:26:27.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.792+0000 7fcb62811800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:27.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.794+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.794+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.798+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.798+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.799+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:d state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.799+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to wf7vo0jv0j3kl90dqu5zmpnvrosshqra 2023-12-03T11:26:27.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.799+0000 7fcb62811800 20 get_obj_state_impl: setting s->olh_tag to vxk73kb46sbdinprdchh8y67gv0m5usx 2023-12-03T11:26:27.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.799+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=vxk73kb46sbdinprdchh8y67gv0m5usx 2023-12-03T11:26:27.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.801+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.804+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.804+0000 7fcb62811800 20 olh_log_entry: epoch=3 op=3 key=d[HXNOeIDu-jpaDRk3uFJjyWaB1.POgx.] 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.804+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:HXNOeIDu-jpaDRk3uFJjyWaB1.POgx._d state=0x562d5000c978 s->prefetch_data=0 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.804+0000 7fcb62811800 10 manifest: total_size = 12 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.805+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17119508451387565447 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.805+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:HXNOeIDu-jpaDRk3uFJjyWaB1.POgx._d state=0x562d5000c978 s->prefetch_data=0 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.805+0000 7fcb62811800 20 prepare_atomic_modification: state is not atomic. state=0x562d5000c978 2023-12-03T11:26:27.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.805+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.807+0000 7fcb62811800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:27.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.807+0000 7fcb62811800 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2023-12-03T11:26:27.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.807+0000 7fcb62811800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:27.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.807+0000 7fcb62811800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:HXNOeIDu-jpaDRk3uFJjyWaB1.POgx._.Ri1_KVj7bSvXASSvmguxu98iMy2mHEC_0 2023-12-03T11:26:27.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.810+0000 7fcb567fc640 20 handle_completion(): completion ok for obj=d[HXNOeIDu-jpaDRk3uFJjyWaB1.POgx.] 2023-12-03T11:26:27.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.813+0000 7fcb62811800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:27.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.815+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.815+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.817+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.817+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.817+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:e state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.818+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to 5hlkx4pwy2wfy3lq8p10f37rgx20355y 2023-12-03T11:26:27.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.818+0000 7fcb62811800 20 get_obj_state_impl: setting s->olh_tag to 9io4gr934viq74b37t20jwtwfythppd0 2023-12-03T11:26:27.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.818+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=9io4gr934viq74b37t20jwtwfythppd0 2023-12-03T11:26:27.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.819+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 olh_log_entry: epoch=2 op=2 key=e[] 2023-12-03T11:26:27.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 olh_log_entry: epoch=2 op=3 key=e[diw8y9dKNp1j7SQEN7z.TZ4xoPKvKxF] 2023-12-03T11:26:27.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:diw8y9dKNp1j7SQEN7z.TZ4xoPKvKxF_e state=0x562d50019838 s->prefetch_data=0 2023-12-03T11:26:27.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 10 manifest: total_size = 11 2023-12-03T11:26:27.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14075897428612979887 2023-12-03T11:26:27.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:diw8y9dKNp1j7SQEN7z.TZ4xoPKvKxF_e state=0x562d50019838 s->prefetch_data=0 2023-12-03T11:26:27.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 prepare_atomic_modification: state is not atomic. state=0x562d50019838 2023-12-03T11:26:27.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.822+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.828+0000 7fcb62811800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:27.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.828+0000 7fcb62811800 20 RGWObjManifest::operator++(): result: ofs=11 stripe_ofs=11 part_ofs=0 rule->part_size=0 2023-12-03T11:26:27.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.828+0000 7fcb62811800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:27.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.828+0000 7fcb62811800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:diw8y9dKNp1j7SQEN7z.TZ4xoPKvKxF_.E_2OFhN1qaXukXcUqjM_WgR5LQFVDxJ_0 2023-12-03T11:26:27.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.829+0000 7fcb56ffd640 20 handle_completion(): completion ok for obj=e[diw8y9dKNp1j7SQEN7z.TZ4xoPKvKxF] 2023-12-03T11:26:27.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.831+0000 7fcb62811800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:27.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.832+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:e state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.832+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=9io4gr934viq74b37t20jwtwfythppd0 2023-12-03T11:26:27.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.839+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.842+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.842+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.843+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:f state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.843+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to 3opcq4lbkroytne52ultkt9p7ysl2n35 2023-12-03T11:26:27.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.843+0000 7fcb62811800 20 get_obj_state_impl: setting s->olh_tag to k0bscavxwl69uxlvbl82qvyv97y2iqyl 2023-12-03T11:26:27.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.843+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=k0bscavxwl69uxlvbl82qvyv97y2iqyl 2023-12-03T11:26:27.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.844+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 olh_log_entry: epoch=2 op=2 key=f[] 2023-12-03T11:26:27.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 olh_log_entry: epoch=2 op=3 key=f[dGu.76LSIQEj6KCWGTQ770W71TFEFMx] 2023-12-03T11:26:27.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:dGu.76LSIQEj6KCWGTQ770W71TFEFMx_f state=0x562d50019838 s->prefetch_data=0 2023-12-03T11:26:27.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 10 manifest: total_size = 12 2023-12-03T11:26:27.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17994489678172248069 2023-12-03T11:26:27.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:_:dGu.76LSIQEj6KCWGTQ770W71TFEFMx_f state=0x562d50019838 s->prefetch_data=0 2023-12-03T11:26:27.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 prepare_atomic_modification: state is not atomic. state=0x562d50019838 2023-12-03T11:26:27.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.850+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.853+0000 7fcb62811800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:26:27.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.853+0000 7fcb62811800 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2023-12-03T11:26:27.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.853+0000 7fcb62811800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:26:27.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.853+0000 7fcb62811800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1__shadow:dGu.76LSIQEj6KCWGTQ770W71TFEFMx_.l00Fyeb9Cha3b-SfcZgwU0-Mvmb0bcw_0 2023-12-03T11:26:27.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.855+0000 7fcb567fc640 20 handle_completion(): completion ok for obj=f[dGu.76LSIQEj6KCWGTQ770W71TFEFMx] 2023-12-03T11:26:27.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.855+0000 7fcb62811800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:26:27.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.857+0000 7fcb62811800 20 get_obj_state: octx=0x7fcb4809b000 obj=check-bucket:f state=0x562d50003ed8 s->prefetch_data=0 2023-12-03T11:26:27.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.857+0000 7fcb62811800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=k0bscavxwl69uxlvbl82qvyv97y2iqyl 2023-12-03T11:26:27.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.865+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.870+0000 7fcb62811800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.1.0 2023-12-03T11:26:27.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.870+0000 7fcb62811800 1 NOTICE: finished shard 0 (4 entries removed) 2023-12-03T11:26:27.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.870+0000 7fcb62811800 1 NOTICE: finished all shards (4 entries removed) 2023-12-03T11:26:27.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.871+0000 7fcb267e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:27.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.873+0000 7fcb62811800 20 remove_watcher() i=0 2023-12-03T11:26:27.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.873+0000 7fcb62811800 2 removed watcher, disabling cache 2023-12-03T11:26:27.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.875+0000 7fcb62811800 20 remove_watcher() i=1 2023-12-03T11:26:27.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.875+0000 7fcb62811800 20 remove_watcher() i=2 2023-12-03T11:26:27.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.877+0000 7fcb62811800 20 remove_watcher() i=3 2023-12-03T11:26:27.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.878+0000 7fcb62811800 20 remove_watcher() i=4 2023-12-03T11:26:27.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.879+0000 7fcb62811800 20 remove_watcher() i=5 2023-12-03T11:26:27.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.882+0000 7fcb62811800 20 remove_watcher() i=6 2023-12-03T11:26:27.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:27.883+0000 7fcb62811800 20 remove_watcher() i=7 2023-12-03T11:26:27.975 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket check-bucket 2023-12-03T11:26:28.002 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:26:28.003 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:26:28.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.016+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 realm 2023-12-03T11:26:28.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.018+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.019+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.019+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.020+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:26:28.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.020+0000 7f3767125800 20 searching for the correct realm 2023-12-03T11:26:28.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.029+0000 7f37635ee640 0 -- 172.21.15.12:0/1497993243 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556ed2e6c9f0 msgr2=0x556ed2e6eeb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:26:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.031+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:26:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 20 zone default found 2023-12-03T11:26:28.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 4 Realm: () 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 20 zonegroup default 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.032+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.033+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.033+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.033+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.033+0000 7f3767125800 20 started sync module instance, tier type = 2023-12-03T11:26:28.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.033+0000 7f3767125800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:26:28.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.038+0000 7f37635ee640 0 -- 172.21.15.12:0/1497993243 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f3758033420 msgr2=0x7f37580338a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=0 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=1 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=2 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=3 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=4 2023-12-03T11:26:28.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=5 2023-12-03T11:26:28.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.048+0000 7f3767125800 20 add_watcher() i=6 2023-12-03T11:26:28.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.050+0000 7f3767125800 20 add_watcher() i=7 2023-12-03T11:26:28.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.050+0000 7f3767125800 2 all 8 watchers are set, enabling cache 2023-12-03T11:26:28.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.052+0000 7f36a3fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:28.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.053+0000 7f3767125800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:26:28.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.053+0000 7f3767125800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:26:28.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3767125800 5 note: GC not initialized 2023-12-03T11:26:28.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3692ffd640 20 reqs_thread_entry: start 2023-12-03T11:26:28.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3767125800 20 init_complete bucket index max shards: 1 2023-12-03T11:26:28.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3767125800 20 Filter name: none 2023-12-03T11:26:28.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3690ff9640 20 reqs_thread_entry: start 2023-12-03T11:26:28.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.059+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:28.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:28.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=190 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.060+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.061+0000 7f3767125800 20 rados_obj.operate() r=0 bl.length=368 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.061+0000 7f3767125800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.061+0000 7f3767125800 20 rados->read ofs=0 len=0 2023-12-03T11:26:28.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.061+0000 7f3767125800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:26:28.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.062+0000 7f36a3fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:26:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.064+0000 7f3767125800 20 remove_watcher() i=0 2023-12-03T11:26:28.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.064+0000 7f3767125800 2 removed watcher, disabling cache 2023-12-03T11:26:28.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.065+0000 7f3767125800 20 remove_watcher() i=1 2023-12-03T11:26:28.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.066+0000 7f3767125800 20 remove_watcher() i=2 2023-12-03T11:26:28.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.067+0000 7f3767125800 20 remove_watcher() i=3 2023-12-03T11:26:28.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.068+0000 7f3767125800 20 remove_watcher() i=4 2023-12-03T11:26:28.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.070+0000 7f3767125800 20 remove_watcher() i=5 2023-12-03T11:26:28.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.071+0000 7f3767125800 20 remove_watcher() i=6 2023-12-03T11:26:28.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:26:28.072+0000 7f3767125800 20 remove_watcher() i=7 2023-12-03T11:26:28.078 INFO:tasks.workunit.client.0.smithi012.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2023-12-03T11:26:28.078 INFO:tasks.workunit.client.0.smithi012.stderr:Deleting bucket check-bucket 2023-12-03T11:26:28.097 INFO:tasks.workunit.client.0.smithi012.stderr:Completed bucket check tests 2023-12-03T11:26:28.098 INFO:tasks.workunit.client.0.smithi012.stdout:connected to http://localhost:80 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ deactivate 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin ']' 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset _OLD_VIRTUAL_PATH 2023-12-03T11:26:28.155 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n '' ']' 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ hash -r 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n '' ']' 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset VIRTUAL_ENV 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' '!' '' = nondestructive ']' 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset -f deactivate 2023-12-03T11:26:28.156 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo OK. 2023-12-03T11:26:28.157 INFO:tasks.workunit.client.0.smithi012.stdout:OK. 2023-12-03T11:26:28.157 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:26:28.157 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:26:28.229 INFO:tasks.workunit:Stopping ['rgw/run-bucket-check.sh'] on client.0... 2023-12-03T11:26:28.229 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:26:28.582 DEBUG:teuthology.parallel:result is None 2023-12-03T11:26:28.583 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:26:28.613 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:26:28.614 DEBUG:teuthology.orchestra.run.smithi012:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:26:28.675 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:26:28.675 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:26:28.686 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:26:28.686 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:26:28.686 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:26:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:26:28.729 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:26:28.729 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:26:28.784 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:26:28.784 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:26:28.839 INFO:tasks.workunit:timeout=3h 2023-12-03T11:26:28.840 INFO:tasks.workunit:cleanup=True 2023-12-03T11:26:28.840 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:26:28.896 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:27:35.674 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 97% (11773/12123) Updating files: 98% (11881/12123) Updating files: 99% (12002/12123) Updating files: 100% (12123/12123) Updating files: 100% (12123/12123), done. 2023-12-03T11:27:36.263 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.264 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:27:36.265 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:27:36.266 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:27:36.321 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:27:36.323 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:27:36.324 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:27:36.374 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:27:36.425 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:27:36.466 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:27:36.467 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:27:36.467 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:27:36.507 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:27:36.510 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:27:36.510 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:27:36.567 INFO:tasks.workunit:Running workunits matching cls/test_cls_lock.sh on client.0... 2023-12-03T11:27:36.568 INFO:tasks.workunit:Running workunit cls/test_cls_lock.sh... 2023-12-03T11:27:36.568 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_lock.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_lock.sh 2023-12-03T11:27:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:27:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:27:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:27:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from ClsLock 2023-12-03T11:27:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestMultiLocking 2023-12-03T11:27:37.935 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:37.935 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: exclusive 2023-12-03T11:27:37.935 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:37.935 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:37.936 INFO:tasks.workunit.client.0.smithi012.stdout: client.5099 expiration=2023-12-03T11:29:37.931054+0000 addr=v1:172.21.15.12:0/2261347211 cookie= 2023-12-03T11:27:37.939 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:37.940 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: shared 2023-12-03T11:27:37.940 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:37.940 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:37.940 INFO:tasks.workunit.client.0.smithi012.stdout: client.5099 expiration=2023-12-03T11:29:37.938163+0000 addr=v1:172.21.15.12:0/2261347211 cookie= 2023-12-03T11:27:37.940 INFO:tasks.workunit.client.0.smithi012.stdout: client.5102 expiration=0.000000 addr=v1:172.21.15.12:0/89053240 cookie= 2023-12-03T11:27:37.941 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:37.941 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: shared 2023-12-03T11:27:37.941 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:37.941 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:37.941 INFO:tasks.workunit.client.0.smithi012.stdout: client.5102 expiration=0.000000 addr=v1:172.21.15.12:0/89053240 cookie= 2023-12-03T11:27:38.907 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestMultiLocking (2271 ms) 2023-12-03T11:27:38.908 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestMeta 2023-12-03T11:27:40.968 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:40.968 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: shared 2023-12-03T11:27:40.968 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:40.968 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:40.968 INFO:tasks.workunit.client.0.smithi012.stdout: client.5295 expiration=0.000000 addr=v1:172.21.15.12:0/3871621206 cookie= 2023-12-03T11:27:40.970 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:40.970 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: exclusive 2023-12-03T11:27:40.970 INFO:tasks.workunit.client.0.smithi012.stdout: tag: new_tag 2023-12-03T11:27:40.970 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:40.970 INFO:tasks.workunit.client.0.smithi012.stdout: client.5289 expiration=0.000000 addr=v1:172.21.15.12:0/3290721867 cookie= 2023-12-03T11:27:41.948 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestMeta (3040 ms) 2023-12-03T11:27:41.948 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestCookie 2023-12-03T11:27:43.969 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:43.969 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: exclusive 2023-12-03T11:27:43.969 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:43.969 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout:lock: mylock 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout: lock_type: shared 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout: tag: 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout: lockers: 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout: client.5117 expiration=0.000000 addr=v1:172.21.15.12:0/877954959 cookie= 2023-12-03T11:27:43.974 INFO:tasks.workunit.client.0.smithi012.stdout: client.5117 expiration=0.000000 addr=v1:172.21.15.12:0/877954959 cookie=new cookie 2023-12-03T11:27:44.958 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestCookie (3010 ms) 2023-12-03T11:27:44.958 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestMultipleLocks 2023-12-03T11:27:47.981 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestMultipleLocks (3024 ms) 2023-12-03T11:27:47.981 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestLockDuration 2023-12-03T11:27:55.890 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestLockDuration (7908 ms) 2023-12-03T11:27:55.891 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestAssertLocked 2023-12-03T11:27:58.920 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestAssertLocked (3030 ms) 2023-12-03T11:27:58.920 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestSetCookie 2023-12-03T11:28:01.933 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestSetCookie (3013 ms) 2023-12-03T11:28:01.933 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestRenew 2023-12-03T11:28:22.914 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestRenew (20981 ms) 2023-12-03T11:28:22.914 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralBasic 2023-12-03T11:28:29.893 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestExclusiveEphemeralBasic (6978 ms) 2023-12-03T11:28:29.893 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealEphemeral 2023-12-03T11:28:37.891 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealEphemeral (7999 ms) 2023-12-03T11:28:37.892 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] ClsLock.TestExclusiveEphemeralStealExclusive 2023-12-03T11:28:45.898 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] ClsLock.TestExclusiveEphemeralStealExclusive (8006 ms) 2023-12-03T11:28:45.899 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from ClsLock (69261 ms total) 2023-12-03T11:28:45.899 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:28:45.899 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:28:45.899 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 11 tests from 1 test suite ran. (69261 ms total) 2023-12-03T11:28:45.899 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 11 tests. 2023-12-03T11:28:45.901 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:28:45.901 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:28:45.977 INFO:tasks.workunit:Running workunits matching cls/test_cls_log.sh on client.0... 2023-12-03T11:28:45.978 INFO:tasks.workunit:Running workunit cls/test_cls_log.sh... 2023-12-03T11:28:45.978 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_log.sh 2023-12-03T11:28:46.050 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:28:46.050 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 4 tests from 1 test suite. 2023-12-03T11:28:46.050 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:28:46.050 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 4 tests from cls_log 2023-12-03T11:28:46.050 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_log.test_log_add_same_time 2023-12-03T11:28:48.925 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_log.test_log_add_same_time (2874 ms) 2023-12-03T11:28:48.925 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_log.test_log_add_different_time 2023-12-03T11:28:51.951 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_log.test_log_add_different_time (3026 ms) 2023-12-03T11:28:51.951 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_log.trim_by_time 2023-12-03T11:28:54.955 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_log.trim_by_time (3003 ms) 2023-12-03T11:28:54.955 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_log.trim_by_marker 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_log.trim_by_marker (3028 ms) 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 4 tests from cls_log (11932 ms total) 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 4 tests from 1 test suite ran. (11932 ms total) 2023-12-03T11:28:57.983 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 4 tests. 2023-12-03T11:28:57.986 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:28:57.986 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:28:58.062 INFO:tasks.workunit:Running workunits matching cls/test_cls_refcount.sh on client.0... 2023-12-03T11:28:58.064 INFO:tasks.workunit:Running workunit cls/test_cls_refcount.sh... 2023-12-03T11:28:58.064 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_refcount.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_refcount.sh 2023-12-03T11:28:58.135 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:28:58.135 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 10 tests from 1 test suite. 2023-12-03T11:28:58.135 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:28:58.135 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 10 tests from cls_refcount 2023-12-03T11:28:58.135 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_implicit 2023-12-03T11:29:01.009 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_implicit (2874 ms) 2023-12-03T11:29:01.009 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_implicit_idempotent 2023-12-03T11:29:04.028 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_implicit_idempotent (3019 ms) 2023-12-03T11:29:04.028 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_put_snap 2023-12-03T11:29:13.086 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_put_snap (9057 ms) 2023-12-03T11:29:13.086 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_explicit 2023-12-03T11:29:16.107 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_explicit (3021 ms) 2023-12-03T11:29:16.107 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.set 2023-12-03T11:29:19.142 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.set (3035 ms) 2023-12-03T11:29:19.142 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_implicit_ec 2023-12-03T11:29:26.198 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_implicit_ec (7057 ms) 2023-12-03T11:29:26.198 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_implicit_idempotent_ec 2023-12-03T11:29:33.210 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_implicit_idempotent_ec (7011 ms) 2023-12-03T11:29:33.210 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_put_snap_ec 2023-12-03T11:29:46.287 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_put_snap_ec (13077 ms) 2023-12-03T11:29:46.287 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.test_explicit_ec 2023-12-03T11:29:53.301 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.test_explicit_ec (7015 ms) 2023-12-03T11:29:53.301 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_refcount.set_ec 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_refcount.set_ec (7026 ms) 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 10 tests from cls_refcount (62192 ms total) 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 10 tests from 1 test suite ran. (62192 ms total) 2023-12-03T11:30:00.328 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 10 tests. 2023-12-03T11:30:00.331 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:30:00.331 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:30:00.407 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw.sh on client.0... 2023-12-03T11:30:00.409 INFO:tasks.workunit:Running workunit cls/test_cls_rgw.sh... 2023-12-03T11:30:00.409 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_rgw.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw.sh 2023-12-03T11:30:00.480 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:30:00.480 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 16 tests from 1 test suite. 2023-12-03T11:30:00.480 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:30:00.481 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 16 tests from cls_rgw 2023-12-03T11:30:02.344 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_basic 2023-12-03T11:30:02.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_basic (47 ms) 2023-12-03T11:30:02.392 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_multiple_obj_writers 2023-12-03T11:30:02.425 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_multiple_obj_writers (34 ms) 2023-12-03T11:30:02.425 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_remove_object 2023-12-03T11:30:02.472 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_remove_object (47 ms) 2023-12-03T11:30:02.472 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_suggest 2023-12-03T11:30:04.005 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_suggest (1533 ms) 2023-12-03T11:30:04.005 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_suggest_complete 2023-12-03T11:30:04.012 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_suggest_complete (7 ms) 2023-12-03T11:30:04.012 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_list 2023-12-03T11:30:04.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_list (23 ms) 2023-12-03T11:30:04.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_list_delimited 2023-12-03T11:30:24.982 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_list_delimited (20947 ms) 2023-12-03T11:30:24.982 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.bi_list 2023-12-03T11:30:25.056 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.bi_list (74 ms) 2023-12-03T11:30:25.056 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.gc_set 2023-12-03T11:30:25.066 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.gc_set (10 ms) 2023-12-03T11:30:25.066 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.gc_list 2023-12-03T11:30:25.076 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.gc_list (10 ms) 2023-12-03T11:30:25.076 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.gc_defer 2023-12-03T11:30:32.913 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.gc_defer (7837 ms) 2023-12-03T11:30:32.913 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.usage_basic 2023-12-03T11:30:32.993 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.usage_basic (80 ms) 2023-12-03T11:30:32.993 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.usage_clear_no_obj 2023-12-03T11:30:32.994 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.usage_clear_no_obj (1 ms) 2023-12-03T11:30:32.994 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.usage_clear 2023-12-03T11:30:33.147 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.usage_clear (153 ms) 2023-12-03T11:30:33.147 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.bi_log_trim 2023-12-03T11:30:33.192 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.bi_log_trim (44 ms) 2023-12-03T11:30:33.192 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw.index_racing_removes 2023-12-03T11:30:33.206 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw.index_racing_removes (15 ms) 2023-12-03T11:30:33.917 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 16 tests from cls_rgw (30862 ms total) 2023-12-03T11:30:33.917 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:30:33.918 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:30:33.918 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 16 tests from 1 test suite ran. (33437 ms total) 2023-12-03T11:30:33.918 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 16 tests. 2023-12-03T11:30:33.922 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:30:33.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:30:33.996 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_gc.sh on client.0... 2023-12-03T11:30:33.998 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_gc.sh... 2023-12-03T11:30:33.998 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_rgw_gc.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_gc.sh 2023-12-03T11:30:34.069 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:30:34.069 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:30:34.070 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:30:34.070 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from cls_rgw_gc 2023-12-03T11:30:34.070 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.init 2023-12-03T11:30:35.943 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.init (1874 ms) 2023-12-03T11:30:35.943 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops1 2023-12-03T11:30:35.965 INFO:tasks.workunit.client.0.smithi012.stderr:[ ] list info tag = chain-0 2023-12-03T11:30:35.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops1 (22 ms) 2023-12-03T11:30:35.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops2 2023-12-03T11:30:35.973 INFO:tasks.workunit.client.0.smithi012.stderr:[ ] next_marker is: = 0/4263 2023-12-03T11:30:35.974 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops2 (9 ms) 2023-12-03T11:30:35.974 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops5 2023-12-03T11:30:35.982 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops5 (8 ms) 2023-12-03T11:30:35.982 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops6 2023-12-03T11:30:36.462 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops6 (479 ms) 2023-12-03T11:30:36.462 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops7 2023-12-03T11:30:36.471 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops7 (10 ms) 2023-12-03T11:30:36.471 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops8 2023-12-03T11:30:36.479 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops8 (8 ms) 2023-12-03T11:30:36.479 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.gc_queue_ops9 2023-12-03T11:30:36.486 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.gc_queue_ops9 (7 ms) 2023-12-03T11:30:36.486 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_gc.finalize 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_gc.finalize (471 ms) 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from cls_rgw_gc (2888 ms total) 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 9 tests from 1 test suite ran. (2888 ms total) 2023-12-03T11:30:36.957 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 9 tests. 2023-12-03T11:30:36.960 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:30:36.960 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:30:37.035 INFO:tasks.workunit:Running workunits matching cls/test_cls_rgw_stats.sh on client.0... 2023-12-03T11:30:37.036 INFO:tasks.workunit:Running workunit cls/test_cls_rgw_stats.sh... 2023-12-03T11:30:37.036 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_rgw_stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rgw_stats.sh 2023-12-03T11:30:37.116 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:30:37.116 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:30:37.116 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 1 test from 1 test suite. 2023-12-03T11:30:37.116 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:30:38.960 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 1 test from cls_rgw_stats 2023-12-03T11:30:38.960 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] cls_rgw_stats.simulate 2023-12-03T11:30:38.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.981+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~DVZEZDQ031BQ.0 size=453493 2023-12-03T11:30:38.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.981+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=DVZEZDQ031BQ tag=CFTL5WG7AJQ0 type=write size=1502069 2023-12-03T11:30:38.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.982+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=7G48AMO17ZSR type=write size=1022096 2023-12-03T11:30:38.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.984+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~L0VJ75YSFWJE.0 size=753279 2023-12-03T11:30:38.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.984+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=L0VJ75YSFWJE tag=B0A20Z1D9UGE type=write size=1801855 2023-12-03T11:30:38.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.987+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.0 size=1048576 2023-12-03T11:30:38.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.988+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.1 size=1048576 2023-12-03T11:30:38.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.989+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.2 size=806152 2023-12-03T11:30:38.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.989+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=T2EBIEUVQ71N tag=RAWBZ9VMH2JL type=write size=3951880 2023-12-03T11:30:38.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.991+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=CC9ZTGHI1M18 type=write size=588487 2023-12-03T11:30:38.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.993+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.0 size=1048576 2023-12-03T11:30:38.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.994+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.1 size=1048576 2023-12-03T11:30:38.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.995+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.2 size=141383 2023-12-03T11:30:38.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.995+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=N1B0HRCTOH1J tag=BR349SFJK4AR type=write size=3287111 2023-12-03T11:30:38.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.998+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~FSSVSFLZUYOA.0 size=1048576 2023-12-03T11:30:39.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.999+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~FSSVSFLZUYOA.1 size=643240 2023-12-03T11:30:39.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:38.999+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=FSSVSFLZUYOA tag=LFDGM04OJTF9 type=write size=2740392 2023-12-03T11:30:39.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.000+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=XOVVF23E41RT type=del size=3181399 2023-12-03T11:30:39.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.002+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=OGE1IMJH92RH type=write size=589660 2023-12-03T11:30:39.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.003+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=TBEAE3DB4RER type=del size=3864130 2023-12-03T11:30:39.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.005+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~88Z1G3211T3Q.0 size=1013150 2023-12-03T11:30:39.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.005+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=88Z1G3211T3Q tag=TABB3S5IFRNF type=write size=2061726 2023-12-03T11:30:39.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.007+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=H0VOXCITNW1N type=del size=1505616 2023-12-03T11:30:39.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.009+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~R5JCQYA16ZLW.0 size=127435 2023-12-03T11:30:39.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.009+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=R5JCQYA16ZLW tag=DUVLWU65S9B6 type=write size=1176011 2023-12-03T11:30:39.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.012+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TH41MZ6W2KWA.0 size=1048576 2023-12-03T11:30:39.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.013+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TH41MZ6W2KWA.1 size=424597 2023-12-03T11:30:39.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.013+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=TH41MZ6W2KWA tag=SR9KD6L9QUSA type=write size=2521749 2023-12-03T11:30:39.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.014+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=R5Y8W18GLXMB type=del size=736498 2023-12-03T11:30:39.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.016+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=YZU8IFHWQ72N type=del size=1975028 2023-12-03T11:30:39.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.017+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~DVZEZDQ031BQ.0 size=453493 2023-12-03T11:30:39.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.019+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=DVZEZDQ031BQ tag=CFTL5WG7AJQ0 type=write size=1502069 2023-12-03T11:30:39.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.022+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.0 size=1048576 2023-12-03T11:30:39.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.023+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.1 size=1048576 2023-12-03T11:30:39.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.024+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.2 size=123564 2023-12-03T11:30:39.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.024+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=GI1L2O6PZ9ES tag=U9KFWSGYBXN1 type=write size=3269292 2023-12-03T11:30:39.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.025+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=7G48AMO17ZSR type=write size=1022096 2023-12-03T11:30:39.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.026+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=38P043PDQGJO type=del size=374174 2023-12-03T11:30:39.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.028+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~L0VJ75YSFWJE.0 size=753279 2023-12-03T11:30:39.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.030+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=L0VJ75YSFWJE tag=B0A20Z1D9UGE type=write size=1801855 2023-12-03T11:30:39.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.031+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=BDG1B5DDZ4OC type=del size=254586 2023-12-03T11:30:39.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.033+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.0 size=1048576 2023-12-03T11:30:39.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.034+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.1 size=1048576 2023-12-03T11:30:39.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.034+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~T2EBIEUVQ71N.2 size=806152 2023-12-03T11:30:39.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.036+0000 7fc553ded780 1 < canceled multipart upload key=I3U08QX8MLBD upload=T2EBIEUVQ71N tag=RAWBZ9VMH2JL type=write size=3951880 2023-12-03T11:30:39.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.038+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~N4IN5BKDQ6LB.0 size=1048576 2023-12-03T11:30:39.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.039+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~N4IN5BKDQ6LB.1 size=454978 2023-12-03T11:30:39.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.039+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=N4IN5BKDQ6LB tag=BVSA0AEOIO7Z type=write size=2552130 2023-12-03T11:30:39.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.040+0000 7fc553ded780 1 < completed write operation key=WLTMDGRBF4C9 tag=CC9ZTGHI1M18 type=write size=588487 2023-12-03T11:30:39.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.043+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~I5MC7FWO5G3D.0 size=637948 2023-12-03T11:30:39.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.043+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=I5MC7FWO5G3D tag=UP2QKST554GY type=write size=1686524 2023-12-03T11:30:39.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.044+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.0 size=1048576 2023-12-03T11:30:39.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.045+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.1 size=1048576 2023-12-03T11:30:39.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.046+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~N1B0HRCTOH1J.2 size=141383 2023-12-03T11:30:39.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.048+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=N1B0HRCTOH1J tag=BR349SFJK4AR type=write size=3287111 2023-12-03T11:30:39.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.050+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=1ZCMTFJ7ZYYM type=del size=702129 2023-12-03T11:30:39.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.050+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~FSSVSFLZUYOA.0 size=1048576 2023-12-03T11:30:39.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.050+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~FSSVSFLZUYOA.1 size=643240 2023-12-03T11:30:39.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.052+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=FSSVSFLZUYOA tag=LFDGM04OJTF9 type=write size=2740392 2023-12-03T11:30:39.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.053+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=7E04WLJX1ZFY type=del size=3723538 2023-12-03T11:30:39.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.055+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=XOVVF23E41RT type=del 2023-12-03T11:30:39.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.056+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=3344U3545RHA type=del size=2505220 2023-12-03T11:30:39.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.058+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=OGE1IMJH92RH type=write size=589660 2023-12-03T11:30:39.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.059+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=16KWR840XXKO type=del size=717541 2023-12-03T11:30:39.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.060+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=TBEAE3DB4RER type=del 2023-12-03T11:30:39.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.062+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~32NMQT3BZUHX.0 size=997406 2023-12-03T11:30:39.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.062+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=32NMQT3BZUHX tag=ZKAP4H88ED1F type=write size=2045982 2023-12-03T11:30:39.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.063+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~88Z1G3211T3Q.0 size=1013150 2023-12-03T11:30:39.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.066+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=88Z1G3211T3Q tag=TABB3S5IFRNF type=write size=2061726 2023-12-03T11:30:39.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.067+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=YLPSTT2WO82M type=del size=4058070 2023-12-03T11:30:39.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.068+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=H0VOXCITNW1N type=del 2023-12-03T11:30:39.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.069+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=134RH4MX9GED type=del size=646712 2023-12-03T11:30:39.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.071+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~R5JCQYA16ZLW.0 size=127435 2023-12-03T11:30:39.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.073+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=R5JCQYA16ZLW tag=DUVLWU65S9B6 type=write size=1176011 2023-12-03T11:30:39.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.075+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.0 size=1048576 2023-12-03T11:30:39.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.076+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.1 size=1048576 2023-12-03T11:30:39.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.077+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.2 size=510668 2023-12-03T11:30:39.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.077+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=R7XRB6M9HLYP tag=TU2IUQ02TLTC type=write size=3656396 2023-12-03T11:30:39.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.078+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~TH41MZ6W2KWA.0 size=1048576 2023-12-03T11:30:39.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.078+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~TH41MZ6W2KWA.1 size=424597 2023-12-03T11:30:39.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.080+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=TH41MZ6W2KWA tag=SR9KD6L9QUSA type=write size=2521749 2023-12-03T11:30:39.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.082+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=JN3DU3EK51UC type=write size=767711 2023-12-03T11:30:39.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.084+0000 7fc553ded780 1 < suggested remove operation key=8GRA2MUBGWJI tag=R5Y8W18GLXMB type=del 2023-12-03T11:30:39.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.085+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=VKIF9YMV6PSF type=del size=797962 2023-12-03T11:30:39.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.087+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=YZU8IFHWQ72N type=del 2023-12-03T11:30:39.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.089+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=UKEBLWBC4R1X type=del size=2172926 2023-12-03T11:30:39.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.090+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.0 size=1048576 2023-12-03T11:30:39.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.090+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.1 size=1048576 2023-12-03T11:30:39.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.091+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~GI1L2O6PZ9ES.2 size=123564 2023-12-03T11:30:39.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.093+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=GI1L2O6PZ9ES tag=U9KFWSGYBXN1 type=write size=3269292 2023-12-03T11:30:39.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.094+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=NBQY85AOG6VD type=del size=434515 2023-12-03T11:30:39.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.096+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=38P043PDQGJO type=del 2023-12-03T11:30:39.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.097+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=LXP2U3A1B3H4 type=del size=3899598 2023-12-03T11:30:39.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.098+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=BDG1B5DDZ4OC type=del 2023-12-03T11:30:39.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.100+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=WQ2JQARP5LJU type=del size=2685115 2023-12-03T11:30:39.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.101+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~N4IN5BKDQ6LB.0 size=1048576 2023-12-03T11:30:39.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.102+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~N4IN5BKDQ6LB.1 size=454978 2023-12-03T11:30:39.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.104+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=N4IN5BKDQ6LB tag=BVSA0AEOIO7Z type=write size=2552130 2023-12-03T11:30:39.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.106+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~C1NNOA7CMXZF.0 size=405551 2023-12-03T11:30:39.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.106+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=C1NNOA7CMXZF tag=BLJ7FC485YC3 type=write size=1454127 2023-12-03T11:30:39.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.107+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~I5MC7FWO5G3D.0 size=637948 2023-12-03T11:30:39.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.110+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=I5MC7FWO5G3D tag=UP2QKST554GY type=write size=1686524 2023-12-03T11:30:39.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.112+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=NONVEYTABZ4B type=del size=1012740 2023-12-03T11:30:39.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.114+0000 7fc553ded780 1 < suggested remove operation key=FP2UZMDKYTD1 tag=1ZCMTFJ7ZYYM type=del 2023-12-03T11:30:39.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.115+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=X5FT0ACF4OLX type=del size=4034047 2023-12-03T11:30:39.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.118+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=7E04WLJX1ZFY type=del 2023-12-03T11:30:39.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.120+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=VUVORG436I27 type=del size=3317146 2023-12-03T11:30:39.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.122+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=3344U3545RHA type=del size=2505220 2023-12-03T11:30:39.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.123+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=KP354WLZCK7U type=write size=122716 2023-12-03T11:30:39.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.125+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=16KWR840XXKO type=del 2023-12-03T11:30:39.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.126+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=DPZX9E3D6097 type=write size=304654 2023-12-03T11:30:39.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.127+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~32NMQT3BZUHX.0 size=997406 2023-12-03T11:30:39.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.129+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=32NMQT3BZUHX tag=ZKAP4H88ED1F type=write size=2045982 2023-12-03T11:30:39.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.131+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.0 size=1048576 2023-12-03T11:30:39.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.132+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.1 size=1048576 2023-12-03T11:30:39.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.133+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.2 size=567203 2023-12-03T11:30:39.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.133+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=Y1VAPJBB8DLE tag=6ZZAMOGRGQ5P type=write size=3712931 2023-12-03T11:30:39.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.134+0000 7fc553ded780 1 < canceled operation key=UKAL8FCVIRSH tag=YLPSTT2WO82M type=del size=4058070 2023-12-03T11:30:39.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.137+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~7OW4J9D689V9.0 size=1048576 2023-12-03T11:30:39.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.137+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~7OW4J9D689V9.1 size=660860 2023-12-03T11:30:39.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.137+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=7OW4J9D689V9 tag=MHNGZ9K9H6U7 type=write size=2758012 2023-12-03T11:30:39.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.139+0000 7fc553ded780 1 < canceled operation key=ICDF1EYNI5IU tag=134RH4MX9GED type=del size=646712 2023-12-03T11:30:39.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.140+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=8BQVMLOL2Y7O type=del size=2713856 2023-12-03T11:30:39.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.141+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.0 size=1048576 2023-12-03T11:30:39.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.143+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.1 size=1048576 2023-12-03T11:30:39.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.144+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R7XRB6M9HLYP.2 size=510668 2023-12-03T11:30:39.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.146+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=R7XRB6M9HLYP tag=TU2IUQ02TLTC type=write size=3656396 2023-12-03T11:30:39.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.147+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=N4EQXWRBEPTI type=del size=3010833 2023-12-03T11:30:39.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.149+0000 7fc553ded780 1 < suggested update operation key=TQRZT37AB9WC tag=JN3DU3EK51UC type=write size=767711 2023-12-03T11:30:39.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.151+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=WAWYY3983UBE type=write size=86711 2023-12-03T11:30:39.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.152+0000 7fc553ded780 -1 < failed to complete operation key=CZP8OQRU641T tag=VKIF9YMV6PSF type=del size=797962: (22) Invalid argument 2023-12-03T11:30:39.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.154+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CB7T57W6JJWM.0 size=767290 2023-12-03T11:30:39.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.154+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=CB7T57W6JJWM tag=UCNH2WBHX1XS type=write size=1815866 2023-12-03T11:30:39.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.156+0000 7fc553ded780 1 < suggested update operation key=PL22GPUFN3O0 tag=UKEBLWBC4R1X type=del size=2172926 2023-12-03T11:30:39.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.157+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=R4TXTBX7TRKH type=del size=955653 2023-12-03T11:30:39.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.159+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=NBQY85AOG6VD type=del 2023-12-03T11:30:39.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.160+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=O1VHVDOFFDD0 type=write size=901319 2023-12-03T11:30:39.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.161+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=LXP2U3A1B3H4 type=del 2023-12-03T11:30:39.164 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.163+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~EYFQBRYVHWGD.0 size=1048576 2023-12-03T11:30:39.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.164+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~EYFQBRYVHWGD.1 size=504538 2023-12-03T11:30:39.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.164+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=EYFQBRYVHWGD tag=B0YW3RNNFLEK type=write size=2601690 2023-12-03T11:30:39.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.166+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=WQ2JQARP5LJU type=del 2023-12-03T11:30:39.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.167+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=449Y1IF95002 type=del size=3760559 2023-12-03T11:30:39.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.168+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~C1NNOA7CMXZF.0 size=405551 2023-12-03T11:30:39.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.170+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=C1NNOA7CMXZF tag=BLJ7FC485YC3 type=write size=1454127 2023-12-03T11:30:39.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.172+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~25BKCD3O6EI8.0 size=1048576 2023-12-03T11:30:39.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.173+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~25BKCD3O6EI8.1 size=283833 2023-12-03T11:30:39.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.173+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=25BKCD3O6EI8 tag=16QFI8KOTJQM type=write size=2380985 2023-12-03T11:30:39.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.176+0000 7fc553ded780 1 < suggested remove operation key=EK7V6U9DSLD0 tag=NONVEYTABZ4B type=del 2023-12-03T11:30:39.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.178+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.0 size=1048576 2023-12-03T11:30:39.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.179+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.1 size=1048576 2023-12-03T11:30:39.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.180+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.2 size=367167 2023-12-03T11:30:39.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.180+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=T6Q4B63GMUZE tag=QU3OB4BGDJT2 type=write size=3512895 2023-12-03T11:30:39.182 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.181+0000 7fc553ded780 1 < canceled operation key=J664UUE697Q8 tag=X5FT0ACF4OLX type=del size=4034047 2023-12-03T11:30:39.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.184+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.0 size=1048576 2023-12-03T11:30:39.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.185+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.1 size=1048576 2023-12-03T11:30:39.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.186+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.2 size=754648 2023-12-03T11:30:39.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.186+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=XC2X6RVHNM0G tag=220GZL372HNC type=write size=3900376 2023-12-03T11:30:39.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.187+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=VUVORG436I27 type=del 2023-12-03T11:30:39.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.189+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~2H11G3MP0A8E.0 size=1048576 2023-12-03T11:30:39.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.190+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~2H11G3MP0A8E.1 size=171193 2023-12-03T11:30:39.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.190+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=2H11G3MP0A8E tag=KH87PW0DMGBQ type=write size=2268345 2023-12-03T11:30:39.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.192+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=KP354WLZCK7U type=write size=122716 2023-12-03T11:30:39.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.193+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=6BYVVUKC4XC9 type=del size=3433275 2023-12-03T11:30:39.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.195+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=DPZX9E3D6097 type=write size=304654 2023-12-03T11:30:39.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.196+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=T6OEJO38POJB type=write size=215831 2023-12-03T11:30:39.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.197+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.0 size=1048576 2023-12-03T11:30:39.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.198+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.1 size=1048576 2023-12-03T11:30:39.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.199+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Y1VAPJBB8DLE.2 size=567203 2023-12-03T11:30:39.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.201+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=Y1VAPJBB8DLE tag=6ZZAMOGRGQ5P type=write size=3712931 2023-12-03T11:30:39.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.205+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~GQA94HGFEAE1.0 size=1048576 2023-12-03T11:30:39.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.206+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~GQA94HGFEAE1.1 size=608069 2023-12-03T11:30:39.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.206+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=GQA94HGFEAE1 tag=K8VKEUUW4SVY type=write size=2705221 2023-12-03T11:30:39.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.207+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~7OW4J9D689V9.0 size=1048576 2023-12-03T11:30:39.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.208+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~7OW4J9D689V9.1 size=660860 2023-12-03T11:30:39.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.210+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=7OW4J9D689V9 tag=MHNGZ9K9H6U7 type=write size=2758012 2023-12-03T11:30:39.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.211+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=1R3FB3CQCXL4 type=del size=3693808 2023-12-03T11:30:39.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.212+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=8BQVMLOL2Y7O type=del 2023-12-03T11:30:39.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.214+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=L0XNSODA9AL9 type=del size=2475261 2023-12-03T11:30:39.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.216+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=N4EQXWRBEPTI type=del 2023-12-03T11:30:39.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.217+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=OQV7MLB6CSDT type=del size=3920009 2023-12-03T11:30:39.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.220+0000 7fc553ded780 1 < suggested update operation key=J664UUE697Q8 tag=WAWYY3983UBE type=write size=86711 2023-12-03T11:30:39.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.222+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.0 size=1048576 2023-12-03T11:30:39.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.223+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.1 size=1048576 2023-12-03T11:30:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.224+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.2 size=184633 2023-12-03T11:30:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.224+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=K0SABUA18BDJ tag=MRRXTIH6UE12 type=write size=3330361 2023-12-03T11:30:39.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.224+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~CB7T57W6JJWM.0 size=767290 2023-12-03T11:30:39.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.227+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=CB7T57W6JJWM tag=UCNH2WBHX1XS type=write size=1815866 2023-12-03T11:30:39.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.229+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=75JFSY1NBQXF type=del size=2661809 2023-12-03T11:30:39.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.230+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=R4TXTBX7TRKH type=del 2023-12-03T11:30:39.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.231+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=DJYOFK2R9X3X type=del size=74902 2023-12-03T11:30:39.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.233+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=O1VHVDOFFDD0 type=write size=901319 2023-12-03T11:30:39.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.235+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~Q4263BNTY1YO.0 size=882753 2023-12-03T11:30:39.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.235+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=Q4263BNTY1YO tag=89GXEVTT4EDW type=write size=1931329 2023-12-03T11:30:39.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.236+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~EYFQBRYVHWGD.0 size=1048576 2023-12-03T11:30:39.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.237+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~EYFQBRYVHWGD.1 size=504538 2023-12-03T11:30:39.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.239+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=EYFQBRYVHWGD tag=B0YW3RNNFLEK type=write size=2601690 2023-12-03T11:30:39.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.241+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=YX2DJP8CVLNS type=write size=106693 2023-12-03T11:30:39.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.242+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=449Y1IF95002 type=del 2023-12-03T11:30:39.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.245+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.0 size=1048576 2023-12-03T11:30:39.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.246+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.1 size=1048576 2023-12-03T11:30:39.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.247+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.2 size=97906 2023-12-03T11:30:39.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.247+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=1VVK21WMHACZ tag=9SB5BBPXZOHN type=write size=3243634 2023-12-03T11:30:39.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.248+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~25BKCD3O6EI8.0 size=1048576 2023-12-03T11:30:39.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.250+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~25BKCD3O6EI8.1 size=283833 2023-12-03T11:30:39.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.252+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=25BKCD3O6EI8 tag=16QFI8KOTJQM type=write size=2380985 2023-12-03T11:30:39.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.254+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.0 size=1048576 2023-12-03T11:30:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.255+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.1 size=1048576 2023-12-03T11:30:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.256+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.2 size=562924 2023-12-03T11:30:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.256+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=18QH2SCSO816 tag=H0BYNHSQ9A19 type=write size=3708652 2023-12-03T11:30:39.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.258+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.0 size=1048576 2023-12-03T11:30:39.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.259+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.1 size=1048576 2023-12-03T11:30:39.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.260+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~T6Q4B63GMUZE.2 size=367167 2023-12-03T11:30:39.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.262+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=T6Q4B63GMUZE tag=QU3OB4BGDJT2 type=write size=3512895 2023-12-03T11:30:39.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.263+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=A1H3RNLW36KG type=del size=2960931 2023-12-03T11:30:39.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.265+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.0 size=1048576 2023-12-03T11:30:39.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.265+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.1 size=1048576 2023-12-03T11:30:39.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.265+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~XC2X6RVHNM0G.2 size=754648 2023-12-03T11:30:39.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.268+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=XC2X6RVHNM0G tag=220GZL372HNC type=write size=3900376 2023-12-03T11:30:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.269+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=0I16AX4G08ZN type=del size=34758 2023-12-03T11:30:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.269+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~2H11G3MP0A8E.0 size=1048576 2023-12-03T11:30:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.270+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~2H11G3MP0A8E.1 size=171193 2023-12-03T11:30:39.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.272+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=2H11G3MP0A8E tag=KH87PW0DMGBQ type=write size=2268345 2023-12-03T11:30:39.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.273+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=RMFFI55OV9FM type=del size=2767449 2023-12-03T11:30:39.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.274+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=6BYVVUKC4XC9 type=del 2023-12-03T11:30:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.277+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~1OO7WNGPECG9.0 size=1048576 2023-12-03T11:30:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.278+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~1OO7WNGPECG9.1 size=637846 2023-12-03T11:30:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.278+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=1OO7WNGPECG9 tag=VC8Y6M33FJ4T type=write size=2734998 2023-12-03T11:30:39.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.279+0000 7fc553ded780 -1 < failed to complete operation key=PL22GPUFN3O0 tag=T6OEJO38POJB type=write size=215831: (22) Invalid argument 2023-12-03T11:30:39.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.282+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.0 size=1048576 2023-12-03T11:30:39.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.283+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.1 size=1048576 2023-12-03T11:30:39.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.283+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.2 size=786221 2023-12-03T11:30:39.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.283+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=Y1UQW1XJS3LR tag=E8SJI0UDNSA4 type=write size=3931949 2023-12-03T11:30:39.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.285+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~GQA94HGFEAE1.0 size=1048576 2023-12-03T11:30:39.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.285+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~GQA94HGFEAE1.1 size=608069 2023-12-03T11:30:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.287+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=GQA94HGFEAE1 tag=K8VKEUUW4SVY type=write size=2705221 2023-12-03T11:30:39.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.288+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=IB6PA23M3ENN type=del size=810098 2023-12-03T11:30:39.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.289+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=1R3FB3CQCXL4 type=del 2023-12-03T11:30:39.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.291+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=XR8U89GVM3N2 type=del size=196979 2023-12-03T11:30:39.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.292+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=L0XNSODA9AL9 type=del 2023-12-03T11:30:39.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.293+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=7YUR657HM1U7 type=del size=4183242 2023-12-03T11:30:39.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.294+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=OQV7MLB6CSDT type=del 2023-12-03T11:30:39.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.296+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=8T7IFMM2CZUY type=del size=2112553 2023-12-03T11:30:39.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.297+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.0 size=1048576 2023-12-03T11:30:39.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.298+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.1 size=1048576 2023-12-03T11:30:39.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.299+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~K0SABUA18BDJ.2 size=184633 2023-12-03T11:30:39.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.301+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=K0SABUA18BDJ tag=MRRXTIH6UE12 type=write size=3330361 2023-12-03T11:30:39.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.303+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~G1N8AA3P939F.0 size=1048576 2023-12-03T11:30:39.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.304+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~G1N8AA3P939F.1 size=497460 2023-12-03T11:30:39.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.304+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=G1N8AA3P939F tag=Y9C4LNC37G77 type=write size=2594612 2023-12-03T11:30:39.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.306+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=75JFSY1NBQXF type=del 2023-12-03T11:30:39.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.307+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=V6QFPXDZ4LRI type=write size=1041754 2023-12-03T11:30:39.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.310+0000 7fc553ded780 1 < suggested remove operation key=C2TPF9Q5Z7XQ tag=DJYOFK2R9X3X type=del 2023-12-03T11:30:39.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.311+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=ODUO98NZL240 type=del size=1502935 2023-12-03T11:30:39.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.312+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~Q4263BNTY1YO.0 size=882753 2023-12-03T11:30:39.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.314+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=Q4263BNTY1YO tag=89GXEVTT4EDW type=write size=1931329 2023-12-03T11:30:39.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.316+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~J6LICAUV6I45.0 size=1048576 2023-12-03T11:30:39.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.318+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~J6LICAUV6I45.1 size=809965 2023-12-03T11:30:39.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.318+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=J6LICAUV6I45 tag=FQE2PNCYHTQZ type=write size=2907117 2023-12-03T11:30:39.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.319+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=YX2DJP8CVLNS type=write size=106693 2023-12-03T11:30:39.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.320+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=7R6MWIMK56T2 type=del size=3044473 2023-12-03T11:30:39.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.321+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.0 size=1048576 2023-12-03T11:30:39.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.322+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.1 size=1048576 2023-12-03T11:30:39.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.323+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~1VVK21WMHACZ.2 size=97906 2023-12-03T11:30:39.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.325+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=1VVK21WMHACZ tag=9SB5BBPXZOHN type=write size=3243634 2023-12-03T11:30:39.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.327+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=UX6J3XIHJIFA type=del size=817535 2023-12-03T11:30:39.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.328+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.0 size=1048576 2023-12-03T11:30:39.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.329+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.1 size=1048576 2023-12-03T11:30:39.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.330+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~18QH2SCSO816.2 size=562924 2023-12-03T11:30:39.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.332+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=18QH2SCSO816 tag=H0BYNHSQ9A19 type=write size=3708652 2023-12-03T11:30:39.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.333+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=JM2ZBNQ1YCL2 type=del size=3970235 2023-12-03T11:30:39.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.335+0000 7fc553ded780 1 < canceled operation key=FP2UZMDKYTD1 tag=A1H3RNLW36KG type=del size=2960931 2023-12-03T11:30:39.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.337+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~LPV5CMTU3YZM.0 size=1044761 2023-12-03T11:30:39.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.337+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=LPV5CMTU3YZM tag=831VYV2IMGIF type=write size=2093337 2023-12-03T11:30:39.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.338+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=0I16AX4G08ZN type=del 2023-12-03T11:30:39.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.339+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=YOY3MZUG28GX type=del size=393403 2023-12-03T11:30:39.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.341+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=RMFFI55OV9FM type=del 2023-12-03T11:30:39.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.343+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~KD09R2U5D8HB.0 size=1048576 2023-12-03T11:30:39.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.344+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~KD09R2U5D8HB.1 size=626143 2023-12-03T11:30:39.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.344+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=KD09R2U5D8HB tag=1TD92QQPLBR8 type=write size=2723295 2023-12-03T11:30:39.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.346+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~1OO7WNGPECG9.0 size=1048576 2023-12-03T11:30:39.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.347+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~1OO7WNGPECG9.1 size=637846 2023-12-03T11:30:39.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.349+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=1OO7WNGPECG9 tag=VC8Y6M33FJ4T type=write size=2734998 2023-12-03T11:30:39.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.350+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=2VE6R860SKCN type=del size=2575135 2023-12-03T11:30:39.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.351+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.0 size=1048576 2023-12-03T11:30:39.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.352+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.1 size=1048576 2023-12-03T11:30:39.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.353+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Y1UQW1XJS3LR.2 size=786221 2023-12-03T11:30:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.355+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=Y1UQW1XJS3LR tag=E8SJI0UDNSA4 type=write size=3931949 2023-12-03T11:30:39.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.357+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=WVM0GXXLVJ7M type=del size=2614459 2023-12-03T11:30:39.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.358+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=IB6PA23M3ENN type=del 2023-12-03T11:30:39.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.360+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~H7JTWUB2QE16.0 size=525060 2023-12-03T11:30:39.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.360+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=H7JTWUB2QE16 tag=IP8EUN3OFRRP type=write size=1573636 2023-12-03T11:30:39.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.361+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=XR8U89GVM3N2 type=del 2023-12-03T11:30:39.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.362+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=AZW0XI5U0FF3 type=del size=3685504 2023-12-03T11:30:39.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.364+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=7YUR657HM1U7 type=del 2023-12-03T11:30:39.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.366+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.0 size=1048576 2023-12-03T11:30:39.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.367+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.1 size=1048576 2023-12-03T11:30:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.368+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.2 size=377157 2023-12-03T11:30:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.368+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=CPY2GZG4OU15 tag=HMT1T1BV3VNO type=write size=3522885 2023-12-03T11:30:39.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.369+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=8T7IFMM2CZUY type=del 2023-12-03T11:30:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.370+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=101FUNCG9MBB type=del size=67406 2023-12-03T11:30:39.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.372+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~G1N8AA3P939F.0 size=1048576 2023-12-03T11:30:39.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.373+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~G1N8AA3P939F.1 size=497460 2023-12-03T11:30:39.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.375+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=G1N8AA3P939F tag=Y9C4LNC37G77 type=write size=2594612 2023-12-03T11:30:39.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.376+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=BCYFA5UBI58E type=del size=1998193 2023-12-03T11:30:39.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.377+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=V6QFPXDZ4LRI type=write size=1041754 2023-12-03T11:30:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.379+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~20LQ4ZHYMJ8U.0 size=370595 2023-12-03T11:30:39.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.379+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=20LQ4ZHYMJ8U tag=JSBQHDJU5T4F type=write size=1419171 2023-12-03T11:30:39.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.381+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=ODUO98NZL240 type=del 2023-12-03T11:30:39.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.382+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=C673SPOO84DO type=del size=2354902 2023-12-03T11:30:39.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.383+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~J6LICAUV6I45.0 size=1048576 2023-12-03T11:30:39.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.384+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~J6LICAUV6I45.1 size=809965 2023-12-03T11:30:39.387 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.386+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=J6LICAUV6I45 tag=FQE2PNCYHTQZ type=write size=2907117 2023-12-03T11:30:39.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.387+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=EVMQ5QDZBCFM type=del size=3831360 2023-12-03T11:30:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.388+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=7R6MWIMK56T2 type=del 2023-12-03T11:30:39.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.390+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=05JWLZPPNJ6K type=del size=3276799 2023-12-03T11:30:39.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.391+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=UX6J3XIHJIFA type=del 2023-12-03T11:30:39.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.392+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=MJQ23KYS1QZG type=del size=1523437 2023-12-03T11:30:39.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.393+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=JM2ZBNQ1YCL2 type=del 2023-12-03T11:30:39.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.395+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=8W10S2M2NTNV type=del size=1178017 2023-12-03T11:30:39.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.396+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~LPV5CMTU3YZM.0 size=1044761 2023-12-03T11:30:39.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.398+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=LPV5CMTU3YZM tag=831VYV2IMGIF type=write size=2093337 2023-12-03T11:30:39.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.399+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=UU1JZ147NHNC type=del size=2050139 2023-12-03T11:30:39.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.400+0000 7fc553ded780 1 < canceled operation key=L4ME9RYRR4NG tag=YOY3MZUG28GX type=del size=393403 2023-12-03T11:30:39.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.404+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~UHEJ6FD8B0OQ.0 size=1048576 2023-12-03T11:30:39.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.405+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~UHEJ6FD8B0OQ.1 size=418770 2023-12-03T11:30:39.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.405+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=UHEJ6FD8B0OQ tag=Q4A9WX0005SX type=write size=2515922 2023-12-03T11:30:39.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.407+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~KD09R2U5D8HB.0 size=1048576 2023-12-03T11:30:39.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.408+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~KD09R2U5D8HB.1 size=626143 2023-12-03T11:30:39.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.410+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=KD09R2U5D8HB tag=1TD92QQPLBR8 type=write size=2723295 2023-12-03T11:30:39.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.412+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~0ZM56TSGD301.0 size=1048576 2023-12-03T11:30:39.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.413+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~0ZM56TSGD301.1 size=896534 2023-12-03T11:30:39.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.413+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=0ZM56TSGD301 tag=NIYNAWFNHVTB type=write size=2993686 2023-12-03T11:30:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.414+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=2VE6R860SKCN type=del 2023-12-03T11:30:39.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.415+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=8B302RJ0SJWA type=del size=1458412 2023-12-03T11:30:39.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.417+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=WVM0GXXLVJ7M type=del 2023-12-03T11:30:39.419 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.418+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=KLDBZQ4K00XQ type=write size=113853 2023-12-03T11:30:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.419+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~H7JTWUB2QE16.0 size=525060 2023-12-03T11:30:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.421+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=H7JTWUB2QE16 tag=IP8EUN3OFRRP type=write size=1573636 2023-12-03T11:30:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.423+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=HCQXH31STSGB type=del size=4016807 2023-12-03T11:30:39.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.424+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=AZW0XI5U0FF3 type=del 2023-12-03T11:30:39.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.425+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=U0H1DMEY3AKY type=write size=629908 2023-12-03T11:30:39.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.426+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.0 size=1048576 2023-12-03T11:30:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.427+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.1 size=1048576 2023-12-03T11:30:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.428+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~CPY2GZG4OU15.2 size=377157 2023-12-03T11:30:39.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.430+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=CPY2GZG4OU15 tag=HMT1T1BV3VNO type=write size=3522885 2023-12-03T11:30:39.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.432+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=QWCYLVG8FZV6 type=write size=649793 2023-12-03T11:30:39.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.433+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=101FUNCG9MBB type=del 2023-12-03T11:30:39.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.434+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=W1XMWS2R5GZP type=del size=273385 2023-12-03T11:30:39.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.435+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=BCYFA5UBI58E type=del 2023-12-03T11:30:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.437+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~YLWVTDS972AJ.0 size=1048576 2023-12-03T11:30:39.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.439+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~YLWVTDS972AJ.1 size=457866 2023-12-03T11:30:39.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.439+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=YLWVTDS972AJ tag=B8UBBDBGWP81 type=write size=2555018 2023-12-03T11:30:39.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.440+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~20LQ4ZHYMJ8U.0 size=370595 2023-12-03T11:30:39.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.442+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=20LQ4ZHYMJ8U tag=JSBQHDJU5T4F type=write size=1419171 2023-12-03T11:30:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.443+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=DO4DJ9H3YJ8B type=del size=1561297 2023-12-03T11:30:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.444+0000 7fc553ded780 1 < canceled operation key=FP2UZMDKYTD1 tag=C673SPOO84DO type=del size=2354902 2023-12-03T11:30:39.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.446+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~QJHQCOASZR50.0 size=1048576 2023-12-03T11:30:39.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.447+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~QJHQCOASZR50.1 size=509777 2023-12-03T11:30:39.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.447+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=QJHQCOASZR50 tag=0MFKLHN9TH1M type=write size=2606929 2023-12-03T11:30:39.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.449+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=EVMQ5QDZBCFM type=del 2023-12-03T11:30:39.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.450+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=OLZX00ANR264 type=write size=273593 2023-12-03T11:30:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.452+0000 7fc553ded780 1 < suggested update operation key=IRMADF9MXXBZ tag=05JWLZPPNJ6K type=del size=3276799 2023-12-03T11:30:39.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.453+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=V62DTX8QN04E type=write size=303899 2023-12-03T11:30:39.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.455+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=MJQ23KYS1QZG type=del 2023-12-03T11:30:39.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.457+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~JCMZW6AI20P1.0 size=1048576 2023-12-03T11:30:39.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.458+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~JCMZW6AI20P1.1 size=56273 2023-12-03T11:30:39.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.458+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=JCMZW6AI20P1 tag=1OYHTWWX0ZDC type=write size=2153425 2023-12-03T11:30:39.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.460+0000 7fc553ded780 1 < suggested update operation key=CZP8OQRU641T tag=8W10S2M2NTNV type=del size=1178017 2023-12-03T11:30:39.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.463+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.0 size=1048576 2023-12-03T11:30:39.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.464+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.1 size=1048576 2023-12-03T11:30:39.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.465+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.2 size=844450 2023-12-03T11:30:39.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.465+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=6UI9H0HG304T tag=5P7D55LIAGLC type=write size=3990178 2023-12-03T11:30:39.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.467+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=UU1JZ147NHNC type=del 2023-12-03T11:30:39.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.468+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=ZZCP396DB8BU type=del size=3442183 2023-12-03T11:30:39.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.469+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~UHEJ6FD8B0OQ.0 size=1048576 2023-12-03T11:30:39.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.469+0000 7fc553ded780 1 < canceled part key=_multipart_JUYA4LXEQ4US.2~UHEJ6FD8B0OQ.1 size=418770 2023-12-03T11:30:39.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.471+0000 7fc553ded780 1 < canceled multipart upload key=JUYA4LXEQ4US upload=UHEJ6FD8B0OQ tag=Q4A9WX0005SX type=write size=2515922 2023-12-03T11:30:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.472+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=8EKBA310CS7X type=del size=274589 2023-12-03T11:30:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.474+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~0ZM56TSGD301.0 size=1048576 2023-12-03T11:30:39.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.475+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~0ZM56TSGD301.1 size=896534 2023-12-03T11:30:39.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.477+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=0ZM56TSGD301 tag=NIYNAWFNHVTB type=write size=2993686 2023-12-03T11:30:39.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.478+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=PH4TD8URSIGP type=del size=3447680 2023-12-03T11:30:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.480+0000 7fc553ded780 1 < suggested remove operation key=X741JIRT7RIG tag=8B302RJ0SJWA type=del 2023-12-03T11:30:39.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.483+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~I1CWIYAZBZEA.0 size=1048576 2023-12-03T11:30:39.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.484+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~I1CWIYAZBZEA.1 size=886608 2023-12-03T11:30:39.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.484+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=I1CWIYAZBZEA tag=EP0I7QA98EU9 type=write size=2983760 2023-12-03T11:30:39.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.485+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=KLDBZQ4K00XQ type=write size=113853 2023-12-03T11:30:39.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.487+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~FRTYQ4CL7MMM.0 size=455816 2023-12-03T11:30:39.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.487+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=FRTYQ4CL7MMM tag=0RK3JI0RCVZD type=write size=1504392 2023-12-03T11:30:39.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.488+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=HCQXH31STSGB type=del 2023-12-03T11:30:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.490+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=J0Z92TEMRLQ6 type=del size=3669792 2023-12-03T11:30:39.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.491+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=U0H1DMEY3AKY type=write size=629908 2023-12-03T11:30:39.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.493+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.0 size=1048576 2023-12-03T11:30:39.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.494+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.1 size=1048576 2023-12-03T11:30:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.495+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.2 size=161418 2023-12-03T11:30:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.495+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=KOKSHOT11Q9M tag=89HMXFN2X3NX type=write size=3307146 2023-12-03T11:30:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.498+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=QWCYLVG8FZV6 type=write size=649793 2023-12-03T11:30:39.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.499+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=RAJ2RS6XT0H0 type=write size=365743 2023-12-03T11:30:39.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.500+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=W1XMWS2R5GZP type=del 2023-12-03T11:30:39.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.501+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=9NANWITTE3G2 type=write size=557197 2023-12-03T11:30:39.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.503+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~YLWVTDS972AJ.0 size=1048576 2023-12-03T11:30:39.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.503+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~YLWVTDS972AJ.1 size=457866 2023-12-03T11:30:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.504+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=YLWVTDS972AJ tag=B8UBBDBGWP81 type=write size=2555018 2023-12-03T11:30:39.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.506+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=JLGLRKMHP8HI type=write size=853787 2023-12-03T11:30:39.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.507+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=DO4DJ9H3YJ8B type=del 2023-12-03T11:30:39.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.509+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.0 size=1048576 2023-12-03T11:30:39.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.510+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.1 size=1048576 2023-12-03T11:30:39.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.511+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.2 size=745809 2023-12-03T11:30:39.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.511+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=NJWB31M5F35N tag=N0GHF2AV5HOB type=write size=3891537 2023-12-03T11:30:39.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.512+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~QJHQCOASZR50.0 size=1048576 2023-12-03T11:30:39.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.513+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~QJHQCOASZR50.1 size=509777 2023-12-03T11:30:39.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.515+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=QJHQCOASZR50 tag=0MFKLHN9TH1M type=write size=2606929 2023-12-03T11:30:39.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.516+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=FBMHSQTRM66J type=del size=3717367 2023-12-03T11:30:39.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.518+0000 7fc553ded780 1 < completed write operation key=PL22GPUFN3O0 tag=OLZX00ANR264 type=write size=273593 2023-12-03T11:30:39.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.519+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=VXMXBD2U7MJ2 type=write size=192317 2023-12-03T11:30:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.521+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=V62DTX8QN04E type=write size=303899 2023-12-03T11:30:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.523+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~2JSZYYJD189T.0 size=1048576 2023-12-03T11:30:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.524+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~2JSZYYJD189T.1 size=853141 2023-12-03T11:30:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.524+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=2JSZYYJD189T tag=YSBQY3Y489KT type=write size=2950293 2023-12-03T11:30:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.524+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~JCMZW6AI20P1.0 size=1048576 2023-12-03T11:30:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.525+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~JCMZW6AI20P1.1 size=56273 2023-12-03T11:30:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.527+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=JCMZW6AI20P1 tag=1OYHTWWX0ZDC type=write size=2153425 2023-12-03T11:30:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.528+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=B9YKSF9LI4UF type=del size=713745 2023-12-03T11:30:39.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.530+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.0 size=1048576 2023-12-03T11:30:39.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.531+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.1 size=1048576 2023-12-03T11:30:39.533 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.532+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~6UI9H0HG304T.2 size=844450 2023-12-03T11:30:39.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.534+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=6UI9H0HG304T tag=5P7D55LIAGLC type=write size=3990178 2023-12-03T11:30:39.536 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.535+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=H93SYVA45MEB type=del size=2885233 2023-12-03T11:30:39.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.537+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=ZZCP396DB8BU type=del 2023-12-03T11:30:39.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.538+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=FLHV9EYD90JW type=del size=3945590 2023-12-03T11:30:39.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.539+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=8EKBA310CS7X type=del 2023-12-03T11:30:39.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.541+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.0 size=1048576 2023-12-03T11:30:39.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.542+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.1 size=1048576 2023-12-03T11:30:39.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.543+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.2 size=83848 2023-12-03T11:30:39.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.543+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=FFSL9QTQUUMZ tag=NNBQGVE6TC7I type=write size=3229576 2023-12-03T11:30:39.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.545+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=PH4TD8URSIGP type=del 2023-12-03T11:30:39.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.547+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.0 size=1048576 2023-12-03T11:30:39.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.548+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.1 size=1048576 2023-12-03T11:30:39.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.549+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.2 size=583609 2023-12-03T11:30:39.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.549+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=S894RCG3GT3D tag=L9FI3TVNPQRN type=write size=3729337 2023-12-03T11:30:39.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.550+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~I1CWIYAZBZEA.0 size=1048576 2023-12-03T11:30:39.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.551+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~I1CWIYAZBZEA.1 size=886608 2023-12-03T11:30:39.554 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.553+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=I1CWIYAZBZEA tag=EP0I7QA98EU9 type=write size=2983760 2023-12-03T11:30:39.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.555+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=4X6E52Y2VPXM type=del size=4041154 2023-12-03T11:30:39.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.556+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~FRTYQ4CL7MMM.0 size=455816 2023-12-03T11:30:39.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.558+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=FRTYQ4CL7MMM tag=0RK3JI0RCVZD type=write size=1504392 2023-12-03T11:30:39.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.559+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=DFLZD9IK9S4C type=del size=2053504 2023-12-03T11:30:39.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.561+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=J0Z92TEMRLQ6 type=del 2023-12-03T11:30:39.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.563+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~8SP2BYEM4FVX.0 size=1048576 2023-12-03T11:30:39.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.564+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~8SP2BYEM4FVX.1 size=453595 2023-12-03T11:30:39.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.564+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=8SP2BYEM4FVX tag=741FS6E9P3UL type=write size=2550747 2023-12-03T11:30:39.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.565+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.0 size=1048576 2023-12-03T11:30:39.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.566+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.1 size=1048576 2023-12-03T11:30:39.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.567+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~KOKSHOT11Q9M.2 size=161418 2023-12-03T11:30:39.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.569+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=KOKSHOT11Q9M tag=89HMXFN2X3NX type=write size=3307146 2023-12-03T11:30:39.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.570+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=UX5NYQGNL1KU type=write size=264219 2023-12-03T11:30:39.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.572+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=RAJ2RS6XT0H0 type=write size=365743 2023-12-03T11:30:39.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.574+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.0 size=1048576 2023-12-03T11:30:39.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.576+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.1 size=1048576 2023-12-03T11:30:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.577+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.2 size=832373 2023-12-03T11:30:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.577+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=2VA0PXELZAXP tag=643MJ2XC7NPR type=write size=3978101 2023-12-03T11:30:39.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.578+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=9NANWITTE3G2 type=write size=557197 2023-12-03T11:30:39.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.580+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=XQN9RJTC6XKT type=del size=3968624 2023-12-03T11:30:39.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.581+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=JLGLRKMHP8HI type=write size=853787 2023-12-03T11:30:39.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.583+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~QKZAG3PKQK4K.0 size=1048576 2023-12-03T11:30:39.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.584+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~QKZAG3PKQK4K.1 size=1038466 2023-12-03T11:30:39.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.584+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=QKZAG3PKQK4K tag=1Y11VJ5B9BPY type=write size=3135618 2023-12-03T11:30:39.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.585+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.0 size=1048576 2023-12-03T11:30:39.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.586+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.1 size=1048576 2023-12-03T11:30:39.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.587+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NJWB31M5F35N.2 size=745809 2023-12-03T11:30:39.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.589+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=NJWB31M5F35N tag=N0GHF2AV5HOB type=write size=3891537 2023-12-03T11:30:39.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.592+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.0 size=1048576 2023-12-03T11:30:39.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.593+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.1 size=1048576 2023-12-03T11:30:39.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.593+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.2 size=597557 2023-12-03T11:30:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.593+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=7O294C1BT3WV tag=IROKYWJ6P56V type=write size=3743285 2023-12-03T11:30:39.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.595+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=FBMHSQTRM66J type=del 2023-12-03T11:30:39.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.596+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=GEI77IVHQQI1 type=del size=1363833 2023-12-03T11:30:39.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.597+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=VXMXBD2U7MJ2 type=write size=192317 2023-12-03T11:30:39.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.600+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~8I1XN7X3XHJG.0 size=537688 2023-12-03T11:30:39.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.600+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=8I1XN7X3XHJG tag=NQ588N8YC3ZQ type=write size=1586264 2023-12-03T11:30:39.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.601+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~2JSZYYJD189T.0 size=1048576 2023-12-03T11:30:39.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.602+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~2JSZYYJD189T.1 size=853141 2023-12-03T11:30:39.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.604+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=2JSZYYJD189T tag=YSBQY3Y489KT type=write size=2950293 2023-12-03T11:30:39.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.605+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=9OHZ62M20W1Y type=del size=3047809 2023-12-03T11:30:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.607+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=B9YKSF9LI4UF type=del size=713745 2023-12-03T11:30:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.608+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=E41IJGWGQ9JE type=del size=916884 2023-12-03T11:30:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.609+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=H93SYVA45MEB type=del 2023-12-03T11:30:39.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.612+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=23EYG74D5SAE type=del size=3319168 2023-12-03T11:30:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.614+0000 7fc553ded780 1 < suggested remove operation key=588S2ASN9I0W tag=FLHV9EYD90JW type=del 2023-12-03T11:30:39.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.616+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.0 size=1048576 2023-12-03T11:30:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.617+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.1 size=1048576 2023-12-03T11:30:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.618+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.2 size=485428 2023-12-03T11:30:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.618+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=2Z3CMMKYO5G6 tag=81DGRIWWF4XH type=write size=3631156 2023-12-03T11:30:39.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.620+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.0 size=1048576 2023-12-03T11:30:39.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.621+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.1 size=1048576 2023-12-03T11:30:39.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.622+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~FFSL9QTQUUMZ.2 size=83848 2023-12-03T11:30:39.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.624+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=FFSL9QTQUUMZ tag=NNBQGVE6TC7I type=write size=3229576 2023-12-03T11:30:39.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.625+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=E1D7TY3U99PW type=del size=1416719 2023-12-03T11:30:39.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.626+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.0 size=1048576 2023-12-03T11:30:39.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.627+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.1 size=1048576 2023-12-03T11:30:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.628+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~S894RCG3GT3D.2 size=583609 2023-12-03T11:30:39.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.630+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=S894RCG3GT3D tag=L9FI3TVNPQRN type=write size=3729337 2023-12-03T11:30:39.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.631+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=WM7H6678MZSI type=write size=957823 2023-12-03T11:30:39.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.633+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=4X6E52Y2VPXM type=del 2023-12-03T11:30:39.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.634+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=KYZO5VYYP1OY type=del size=697998 2023-12-03T11:30:39.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.635+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=DFLZD9IK9S4C type=del 2023-12-03T11:30:39.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.636+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=5B5BVIW3EWZN type=del size=1729679 2023-12-03T11:30:39.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.637+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~8SP2BYEM4FVX.0 size=1048576 2023-12-03T11:30:39.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.637+0000 7fc553ded780 1 < canceled part key=_multipart_588S2ASN9I0W.2~8SP2BYEM4FVX.1 size=453595 2023-12-03T11:30:39.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.639+0000 7fc553ded780 1 < canceled multipart upload key=588S2ASN9I0W upload=8SP2BYEM4FVX tag=741FS6E9P3UL type=write size=2550747 2023-12-03T11:30:39.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.641+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~5VV4VBGAF5DY.0 size=861260 2023-12-03T11:30:39.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.641+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=5VV4VBGAF5DY tag=UBEHDRCCIRDH type=write size=1909836 2023-12-03T11:30:39.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.643+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=UX5NYQGNL1KU type=write size=264219 2023-12-03T11:30:39.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.644+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=LZ6N05NQ6P4J type=del size=3192655 2023-12-03T11:30:39.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.645+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.0 size=1048576 2023-12-03T11:30:39.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.646+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.1 size=1048576 2023-12-03T11:30:39.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.646+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~2VA0PXELZAXP.2 size=832373 2023-12-03T11:30:39.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.648+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=2VA0PXELZAXP tag=643MJ2XC7NPR type=write size=3978101 2023-12-03T11:30:39.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.650+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~M0CCVX8R5P67.0 size=618855 2023-12-03T11:30:39.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.650+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=M0CCVX8R5P67 tag=A5H74M59ULBZ type=write size=1667431 2023-12-03T11:30:39.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.651+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=XQN9RJTC6XKT type=del 2023-12-03T11:30:39.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.653+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~6JFY1QVIV962.0 size=1048576 2023-12-03T11:30:39.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.654+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~6JFY1QVIV962.1 size=391662 2023-12-03T11:30:39.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.654+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=6JFY1QVIV962 tag=885Y3S7G5PZJ type=write size=2488814 2023-12-03T11:30:39.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.656+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~QKZAG3PKQK4K.0 size=1048576 2023-12-03T11:30:39.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.657+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~QKZAG3PKQK4K.1 size=1038466 2023-12-03T11:30:39.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.659+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=QKZAG3PKQK4K tag=1Y11VJ5B9BPY type=write size=3135618 2023-12-03T11:30:39.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.660+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=DU63D9PENEAD type=del size=1811828 2023-12-03T11:30:39.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.661+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.0 size=1048576 2023-12-03T11:30:39.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.662+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.1 size=1048576 2023-12-03T11:30:39.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.663+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~7O294C1BT3WV.2 size=597557 2023-12-03T11:30:39.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.665+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=7O294C1BT3WV tag=IROKYWJ6P56V type=write size=3743285 2023-12-03T11:30:39.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.667+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=R88JM2AQJ4I4 type=del size=1032828 2023-12-03T11:30:39.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.668+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=GEI77IVHQQI1 type=del 2023-12-03T11:30:39.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.669+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=H7Q85L03XS8N type=del size=3287464 2023-12-03T11:30:39.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.671+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~8I1XN7X3XHJG.0 size=537688 2023-12-03T11:30:39.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.673+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=8I1XN7X3XHJG tag=NQ588N8YC3ZQ type=write size=1586264 2023-12-03T11:30:39.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.674+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=OC0BSQLETJX7 type=del size=642077 2023-12-03T11:30:39.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.675+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=9OHZ62M20W1Y type=del 2023-12-03T11:30:39.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.676+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=I38S4XEF3STI type=del size=800098 2023-12-03T11:30:39.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.677+0000 7fc553ded780 -1 < failed to complete operation key=588S2ASN9I0W tag=E41IJGWGQ9JE type=del size=916884: (22) Invalid argument 2023-12-03T11:30:39.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.679+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=QJZHBCP6OO3Q type=del size=4109152 2023-12-03T11:30:39.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.680+0000 7fc553ded780 -1 < failed to complete operation key=588S2ASN9I0W tag=23EYG74D5SAE type=del size=3319168: (22) Invalid argument 2023-12-03T11:30:39.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.682+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5NW2JKP2FL7Z.0 size=1048576 2023-12-03T11:30:39.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.683+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5NW2JKP2FL7Z.1 size=103873 2023-12-03T11:30:39.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.683+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=5NW2JKP2FL7Z tag=C8JFBQHHC82C type=write size=2201025 2023-12-03T11:30:39.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.684+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.0 size=1048576 2023-12-03T11:30:39.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.685+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.1 size=1048576 2023-12-03T11:30:39.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.686+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Z3CMMKYO5G6.2 size=485428 2023-12-03T11:30:39.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.688+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=2Z3CMMKYO5G6 tag=81DGRIWWF4XH type=write size=3631156 2023-12-03T11:30:39.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.691+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.0 size=1048576 2023-12-03T11:30:39.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.692+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.1 size=1048576 2023-12-03T11:30:39.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.693+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.2 size=33430 2023-12-03T11:30:39.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.693+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=E66QDAMEDWEF tag=WIR3JCGJ0YCS type=write size=3179158 2023-12-03T11:30:39.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.694+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=E1D7TY3U99PW type=del 2023-12-03T11:30:39.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.695+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=2DRPCF71H6HK type=del size=1684664 2023-12-03T11:30:39.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.697+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=WM7H6678MZSI type=write size=957823 2023-12-03T11:30:39.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.698+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=2YBTYJ8FIX5U type=del size=1769449 2023-12-03T11:30:39.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.699+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=KYZO5VYYP1OY type=del 2023-12-03T11:30:39.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.700+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=KGKVQVF8V2IN type=del size=452398 2023-12-03T11:30:39.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.703+0000 7fc553ded780 1 < suggested remove operation key=WLTMDGRBF4C9 tag=5B5BVIW3EWZN type=del 2023-12-03T11:30:39.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.705+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~KO6FVKBXN0CZ.0 size=1048576 2023-12-03T11:30:39.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.706+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~KO6FVKBXN0CZ.1 size=336680 2023-12-03T11:30:39.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.706+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=KO6FVKBXN0CZ tag=I52JHHVNV037 type=write size=2433832 2023-12-03T11:30:39.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.707+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~5VV4VBGAF5DY.0 size=861260 2023-12-03T11:30:39.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.709+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=5VV4VBGAF5DY tag=UBEHDRCCIRDH type=write size=1909836 2023-12-03T11:30:39.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.710+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=WT6L0NGC4BNQ type=write size=290523 2023-12-03T11:30:39.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.712+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=LZ6N05NQ6P4J type=del 2023-12-03T11:30:39.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.713+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=DV5OTOY76B6Q type=del size=3455446 2023-12-03T11:30:39.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.714+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~M0CCVX8R5P67.0 size=618855 2023-12-03T11:30:39.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.716+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=M0CCVX8R5P67 tag=A5H74M59ULBZ type=write size=1667431 2023-12-03T11:30:39.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.718+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.0 size=1048576 2023-12-03T11:30:39.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.719+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.1 size=1048576 2023-12-03T11:30:39.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.720+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.2 size=455431 2023-12-03T11:30:39.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.720+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=X3IAY9Z2F4KV tag=EX8CCEW41CAZ type=write size=3601159 2023-12-03T11:30:39.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.721+0000 7fc553ded780 1 < canceled part key=_multipart_FC4WU6SMNEWV.2~6JFY1QVIV962.0 size=1048576 2023-12-03T11:30:39.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.721+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~6JFY1QVIV962.1 size=391662 2023-12-03T11:30:39.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.724+0000 7fc553ded780 1 < canceled multipart upload key=FC4WU6SMNEWV upload=6JFY1QVIV962 tag=885Y3S7G5PZJ type=write size=2488814 2023-12-03T11:30:39.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.725+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=8E0627KEK63M type=del size=3600145 2023-12-03T11:30:39.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.726+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=DU63D9PENEAD type=del 2023-12-03T11:30:39.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.728+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=E0VKWTGGTJYE type=del size=942948 2023-12-03T11:30:39.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.730+0000 7fc553ded780 1 < suggested update operation key=C2TPF9Q5Z7XQ tag=R88JM2AQJ4I4 type=del size=1032828 2023-12-03T11:30:39.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.731+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=YPX4860VFN4L type=del size=2722639 2023-12-03T11:30:39.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.733+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=H7Q85L03XS8N type=del 2023-12-03T11:30:39.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.735+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~DXVA51NX1H39.0 size=1048576 2023-12-03T11:30:39.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.736+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~DXVA51NX1H39.1 size=101548 2023-12-03T11:30:39.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.736+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=DXVA51NX1H39 tag=86Y03UOXO1SE type=write size=2198700 2023-12-03T11:30:39.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.738+0000 7fc553ded780 1 < suggested remove operation key=TQRZT37AB9WC tag=OC0BSQLETJX7 type=del 2023-12-03T11:30:39.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.740+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=Q6B2LQQ83MG4 type=del size=639427 2023-12-03T11:30:39.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.742+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=I38S4XEF3STI type=del 2023-12-03T11:30:39.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.743+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=K6IQCIV5RSKZ type=write size=482066 2023-12-03T11:30:39.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.744+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=QJZHBCP6OO3Q type=del 2023-12-03T11:30:39.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.746+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=AOQOL3INUJJ1 type=del size=2565402 2023-12-03T11:30:39.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.747+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5NW2JKP2FL7Z.0 size=1048576 2023-12-03T11:30:39.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.748+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5NW2JKP2FL7Z.1 size=103873 2023-12-03T11:30:39.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.750+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=5NW2JKP2FL7Z tag=C8JFBQHHC82C type=write size=2201025 2023-12-03T11:30:39.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.751+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=CVUMK13VZSQK type=del size=1241983 2023-12-03T11:30:39.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.752+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.0 size=1048576 2023-12-03T11:30:39.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.752+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.1 size=1048576 2023-12-03T11:30:39.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.752+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~E66QDAMEDWEF.2 size=33430 2023-12-03T11:30:39.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.754+0000 7fc553ded780 1 < canceled multipart upload key=I3U08QX8MLBD upload=E66QDAMEDWEF tag=WIR3JCGJ0YCS type=write size=3179158 2023-12-03T11:30:39.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.755+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=XGCM7VBONT70 type=write size=62647 2023-12-03T11:30:39.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.758+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=2DRPCF71H6HK type=del 2023-12-03T11:30:39.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.759+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=D9JH3XAB08KL type=write size=195041 2023-12-03T11:30:39.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.760+0000 7fc553ded780 -1 < failed to complete operation key=C2TPF9Q5Z7XQ tag=2YBTYJ8FIX5U type=del size=1769449: (22) Invalid argument 2023-12-03T11:30:39.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.761+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=6AEEHQAC76KS type=del size=2032569 2023-12-03T11:30:39.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.763+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=KGKVQVF8V2IN type=del 2023-12-03T11:30:39.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.765+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~WWPFI04S0FRR.0 size=560968 2023-12-03T11:30:39.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.765+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=WWPFI04S0FRR tag=H13K4UPXOWKB type=write size=1609544 2023-12-03T11:30:39.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.765+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~KO6FVKBXN0CZ.0 size=1048576 2023-12-03T11:30:39.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.766+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~KO6FVKBXN0CZ.1 size=336680 2023-12-03T11:30:39.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.768+0000 7fc553ded780 1 < canceled multipart upload key=I3U08QX8MLBD upload=KO6FVKBXN0CZ tag=I52JHHVNV037 type=write size=2433832 2023-12-03T11:30:39.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.770+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~YN1427P2PEOE.0 size=1048576 2023-12-03T11:30:39.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.771+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~YN1427P2PEOE.1 size=713087 2023-12-03T11:30:39.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.771+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=YN1427P2PEOE tag=8A29Q8WMY7WK type=write size=2810239 2023-12-03T11:30:39.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.773+0000 7fc553ded780 1 < completed write operation key=ZHK2M78A0SLQ tag=WT6L0NGC4BNQ type=write size=290523 2023-12-03T11:30:39.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.775+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~J9AGVI2DHVU4.0 size=1048576 2023-12-03T11:30:39.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.776+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~J9AGVI2DHVU4.1 size=420019 2023-12-03T11:30:39.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.776+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=J9AGVI2DHVU4 tag=RB49YXLL8KAQ type=write size=2517171 2023-12-03T11:30:39.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.777+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=DV5OTOY76B6Q type=del 2023-12-03T11:30:39.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.778+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=YFHN9KH1O4U4 type=del size=4047747 2023-12-03T11:30:39.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.778+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.0 size=1048576 2023-12-03T11:30:39.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.780+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.1 size=1048576 2023-12-03T11:30:39.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.781+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~X3IAY9Z2F4KV.2 size=455431 2023-12-03T11:30:39.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.783+0000 7fc553ded780 1 < canceled multipart upload key=ICDF1EYNI5IU upload=X3IAY9Z2F4KV tag=EX8CCEW41CAZ type=write size=3601159 2023-12-03T11:30:39.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.784+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=4NGR3ANTMXAN type=write size=174006 2023-12-03T11:30:39.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.786+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=8E0627KEK63M type=del 2023-12-03T11:30:39.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.787+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=TGT6Y3ECJ2DL type=del size=782226 2023-12-03T11:30:39.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.788+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=E0VKWTGGTJYE type=del 2023-12-03T11:30:39.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.790+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=R3DOB12M9KYO type=del size=3985579 2023-12-03T11:30:39.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.791+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=YPX4860VFN4L type=del 2023-12-03T11:30:39.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.793+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.0 size=1048576 2023-12-03T11:30:39.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.794+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.1 size=1048576 2023-12-03T11:30:39.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.795+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.2 size=615561 2023-12-03T11:30:39.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.795+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=WKD4F2HDRLNG tag=U0QREFWITQ7W type=write size=3761289 2023-12-03T11:30:39.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.796+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~DXVA51NX1H39.0 size=1048576 2023-12-03T11:30:39.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.797+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~DXVA51NX1H39.1 size=101548 2023-12-03T11:30:39.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.799+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=DXVA51NX1H39 tag=86Y03UOXO1SE type=write size=2198700 2023-12-03T11:30:39.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.801+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.0 size=1048576 2023-12-03T11:30:39.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.802+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.1 size=1048576 2023-12-03T11:30:39.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.803+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.2 size=909348 2023-12-03T11:30:39.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.803+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=Z9N3OLJDZO6D tag=GCBYYXCDCTEM type=write size=4055076 2023-12-03T11:30:39.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.804+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=Q6B2LQQ83MG4 type=del 2023-12-03T11:30:39.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.805+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=YAMBILOG4FOJ type=del size=825129 2023-12-03T11:30:39.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.807+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=K6IQCIV5RSKZ type=write size=482066 2023-12-03T11:30:39.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.810+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~6X7O3SJE6OH5.0 size=1048576 2023-12-03T11:30:39.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.811+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~6X7O3SJE6OH5.1 size=565882 2023-12-03T11:30:39.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.811+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=6X7O3SJE6OH5 tag=GD1KLH5YLGG1 type=write size=2663034 2023-12-03T11:30:39.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.814+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=AOQOL3INUJJ1 type=del 2023-12-03T11:30:39.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.816+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~8LC04T6W7OYM.0 size=1048576 2023-12-03T11:30:39.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.817+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~8LC04T6W7OYM.1 size=714920 2023-12-03T11:30:39.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.817+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=8LC04T6W7OYM tag=NC1M5ITOD77J type=write size=2812072 2023-12-03T11:30:39.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.819+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=CVUMK13VZSQK type=del 2023-12-03T11:30:39.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.821+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.0 size=1048576 2023-12-03T11:30:39.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.822+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.1 size=1048576 2023-12-03T11:30:39.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.823+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.2 size=696357 2023-12-03T11:30:39.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.823+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=TFTFLJTORZLM tag=YUOJLUEPV5SZ type=write size=3842085 2023-12-03T11:30:39.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.824+0000 7fc553ded780 1 < completed write operation key=588S2ASN9I0W tag=XGCM7VBONT70 type=write size=62647 2023-12-03T11:30:39.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.825+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=MJJSP8XX7DMG type=del size=130890 2023-12-03T11:30:39.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.827+0000 7fc553ded780 1 < completed write operation key=WLTMDGRBF4C9 tag=D9JH3XAB08KL type=write size=195041 2023-12-03T11:30:39.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.828+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=BZM9V3Q0IP21 type=write size=805245 2023-12-03T11:30:39.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.829+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=6AEEHQAC76KS type=del 2023-12-03T11:30:39.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.831+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.0 size=1048576 2023-12-03T11:30:39.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.832+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.1 size=1048576 2023-12-03T11:30:39.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.833+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.2 size=887550 2023-12-03T11:30:39.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.833+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=2BR8EWQCDLVL tag=M1KXTW4BENVR type=write size=4033278 2023-12-03T11:30:39.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.834+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~WWPFI04S0FRR.0 size=560968 2023-12-03T11:30:39.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.836+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=WWPFI04S0FRR tag=H13K4UPXOWKB type=write size=1609544 2023-12-03T11:30:39.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.837+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=DLDXPNEQ3328 type=del size=3541172 2023-12-03T11:30:39.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.838+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~YN1427P2PEOE.0 size=1048576 2023-12-03T11:30:39.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.839+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~YN1427P2PEOE.1 size=713087 2023-12-03T11:30:39.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.841+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=YN1427P2PEOE tag=8A29Q8WMY7WK type=write size=2810239 2023-12-03T11:30:39.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.843+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=KBASETW5UXDN type=write size=724619 2023-12-03T11:30:39.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.844+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~J9AGVI2DHVU4.0 size=1048576 2023-12-03T11:30:39.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.844+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~J9AGVI2DHVU4.1 size=420019 2023-12-03T11:30:39.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.846+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=J9AGVI2DHVU4 tag=RB49YXLL8KAQ type=write size=2517171 2023-12-03T11:30:39.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.847+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=CPP0PRNYBCGV type=del size=3617470 2023-12-03T11:30:39.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.849+0000 7fc553ded780 1 < suggested remove operation key=WDV4CW863V33 tag=YFHN9KH1O4U4 type=del 2023-12-03T11:30:39.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.851+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.0 size=1048576 2023-12-03T11:30:39.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.852+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.1 size=1048576 2023-12-03T11:30:39.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.853+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.2 size=961313 2023-12-03T11:30:39.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.853+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=MTAX8K0T3N6K tag=3N99OODT4EL3 type=write size=4107041 2023-12-03T11:30:39.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.854+0000 7fc553ded780 -1 < no bi entry to suggest for operation key=G1OBBD75M5ZQ tag=4NGR3ANTMXAN type=write size=174006: (2) No such file or directory 2023-12-03T11:30:39.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.855+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=L2T1UWT8TWAY type=write size=419509 2023-12-03T11:30:39.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.856+0000 7fc553ded780 -1 < failed to complete operation key=WDV4CW863V33 tag=TGT6Y3ECJ2DL type=del size=782226: (22) Invalid argument 2023-12-03T11:30:39.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.858+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~R0IJWCX3WE3X.0 size=1048576 2023-12-03T11:30:39.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.859+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~R0IJWCX3WE3X.1 size=943229 2023-12-03T11:30:39.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.859+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=R0IJWCX3WE3X tag=KNHPW9H75253 type=write size=3040381 2023-12-03T11:30:39.861 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.860+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=R3DOB12M9KYO type=del 2023-12-03T11:30:39.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.862+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.0 size=1048576 2023-12-03T11:30:39.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.863+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.1 size=1048576 2023-12-03T11:30:39.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.864+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.2 size=1030436 2023-12-03T11:30:39.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.864+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=1MFSZPYZ7P8Z tag=KZTD841QG0Q2 type=write size=4176164 2023-12-03T11:30:39.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.866+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.0 size=1048576 2023-12-03T11:30:39.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.867+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.1 size=1048576 2023-12-03T11:30:39.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.868+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~WKD4F2HDRLNG.2 size=615561 2023-12-03T11:30:39.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.870+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=WKD4F2HDRLNG tag=U0QREFWITQ7W type=write size=3761289 2023-12-03T11:30:39.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.872+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~ON39JI8GIHQH.0 size=1048576 2023-12-03T11:30:39.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.873+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~ON39JI8GIHQH.1 size=1016623 2023-12-03T11:30:39.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.873+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=ON39JI8GIHQH tag=2X2T3Z2UFMPD type=write size=3113775 2023-12-03T11:30:39.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.874+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.0 size=1048576 2023-12-03T11:30:39.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.875+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.1 size=1048576 2023-12-03T11:30:39.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.876+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Z9N3OLJDZO6D.2 size=909348 2023-12-03T11:30:39.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.878+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=Z9N3OLJDZO6D tag=GCBYYXCDCTEM type=write size=4055076 2023-12-03T11:30:39.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.881+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=VOB7XS6R0DA9 type=del size=450583 2023-12-03T11:30:39.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.882+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=YAMBILOG4FOJ type=del size=825129 2023-12-03T11:30:39.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.883+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=23YXZLVCC2GB type=write size=323769 2023-12-03T11:30:39.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.885+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~6X7O3SJE6OH5.0 size=1048576 2023-12-03T11:30:39.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.886+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~6X7O3SJE6OH5.1 size=565882 2023-12-03T11:30:39.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.888+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=6X7O3SJE6OH5 tag=GD1KLH5YLGG1 type=write size=2663034 2023-12-03T11:30:39.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.890+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~IB766LI1CJOW.0 size=1048576 2023-12-03T11:30:39.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.891+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~IB766LI1CJOW.1 size=495535 2023-12-03T11:30:39.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.891+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=IB766LI1CJOW tag=J1C7EALDOA3E type=write size=2592687 2023-12-03T11:30:39.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.892+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~8LC04T6W7OYM.0 size=1048576 2023-12-03T11:30:39.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.893+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~8LC04T6W7OYM.1 size=714920 2023-12-03T11:30:39.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.895+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=8LC04T6W7OYM tag=NC1M5ITOD77J type=write size=2812072 2023-12-03T11:30:39.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.897+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=A45IT67QEVTT type=del size=1375186 2023-12-03T11:30:39.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.898+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.0 size=1048576 2023-12-03T11:30:39.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.899+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.1 size=1048576 2023-12-03T11:30:39.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.900+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~TFTFLJTORZLM.2 size=696357 2023-12-03T11:30:39.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.902+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=TFTFLJTORZLM tag=YUOJLUEPV5SZ type=write size=3842085 2023-12-03T11:30:39.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.904+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~WM9YWYTPF3WO.0 size=1048576 2023-12-03T11:30:39.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.905+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~WM9YWYTPF3WO.1 size=1038806 2023-12-03T11:30:39.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.905+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=WM9YWYTPF3WO tag=NA0A7JXX89KZ type=write size=3135958 2023-12-03T11:30:39.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.907+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=MJJSP8XX7DMG type=del 2023-12-03T11:30:39.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.908+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=Y4N0QBTSBL26 type=del size=887211 2023-12-03T11:30:39.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.910+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=BZM9V3Q0IP21 type=write size=805245 2023-12-03T11:30:39.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.911+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=50U0N49DJATK type=write size=813297 2023-12-03T11:30:39.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.913+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.0 size=1048576 2023-12-03T11:30:39.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.914+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.1 size=1048576 2023-12-03T11:30:39.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.915+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~2BR8EWQCDLVL.2 size=887550 2023-12-03T11:30:39.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.917+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=2BR8EWQCDLVL tag=M1KXTW4BENVR type=write size=4033278 2023-12-03T11:30:39.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.920+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~7RJL2CQLGPUI.0 size=1048576 2023-12-03T11:30:39.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.920+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~7RJL2CQLGPUI.1 size=721109 2023-12-03T11:30:39.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.920+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=7RJL2CQLGPUI tag=4VU7038TDYGJ type=write size=2818261 2023-12-03T11:30:39.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.922+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=DLDXPNEQ3328 type=del 2023-12-03T11:30:39.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.923+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=UH0ZPH0FNOIF type=write size=991332 2023-12-03T11:30:39.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.924+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=KBASETW5UXDN type=write size=724619 2023-12-03T11:30:39.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.925+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=M42KUYTVUBT7 type=del size=1572079 2023-12-03T11:30:39.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.928+0000 7fc553ded780 1 < suggested update operation key=JUDXD9DL4N43 tag=CPP0PRNYBCGV type=del size=3617470 2023-12-03T11:30:39.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.930+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~YEJBKMJYIQXU.0 size=550495 2023-12-03T11:30:39.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.930+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=YEJBKMJYIQXU tag=TJBNSFXF949P type=write size=1599071 2023-12-03T11:30:39.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.931+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.0 size=1048576 2023-12-03T11:30:39.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.932+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.1 size=1048576 2023-12-03T11:30:39.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.933+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~MTAX8K0T3N6K.2 size=961313 2023-12-03T11:30:39.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.935+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=MTAX8K0T3N6K tag=3N99OODT4EL3 type=write size=4107041 2023-12-03T11:30:39.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.936+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=9PYR7912T71R type=write size=34798 2023-12-03T11:30:39.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.938+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=L2T1UWT8TWAY type=write size=419509 2023-12-03T11:30:39.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.940+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~24PMWUN78JJD.0 size=201755 2023-12-03T11:30:39.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.940+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=24PMWUN78JJD tag=HL8SOIZA79PN type=write size=1250331 2023-12-03T11:30:39.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.941+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~R0IJWCX3WE3X.0 size=1048576 2023-12-03T11:30:39.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.942+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~R0IJWCX3WE3X.1 size=943229 2023-12-03T11:30:39.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.944+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=R0IJWCX3WE3X tag=KNHPW9H75253 type=write size=3040381 2023-12-03T11:30:39.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.945+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=XHSJ5Q9Q2KSC type=del size=3297378 2023-12-03T11:30:39.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.947+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.0 size=1048576 2023-12-03T11:30:39.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.948+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.1 size=1048576 2023-12-03T11:30:39.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.949+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~1MFSZPYZ7P8Z.2 size=1030436 2023-12-03T11:30:39.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.952+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=1MFSZPYZ7P8Z tag=KZTD841QG0Q2 type=write size=4176164 2023-12-03T11:30:39.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.953+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=IKHA8JCZJMLU type=del size=436679 2023-12-03T11:30:39.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.955+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~ON39JI8GIHQH.0 size=1048576 2023-12-03T11:30:39.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.956+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~ON39JI8GIHQH.1 size=1016623 2023-12-03T11:30:39.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.962+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=ON39JI8GIHQH tag=2X2T3Z2UFMPD type=write size=3113775 2023-12-03T11:30:39.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.963+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=330MRNFE7ZS1 type=del size=2285232 2023-12-03T11:30:39.967 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.966+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=VOB7XS6R0DA9 type=del 2023-12-03T11:30:39.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.968+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.0 size=1048576 2023-12-03T11:30:39.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.969+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.1 size=1048576 2023-12-03T11:30:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.970+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.2 size=180651 2023-12-03T11:30:39.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.970+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=C1GUKXYMQN1Z tag=H0HKYED50T3A type=write size=3326379 2023-12-03T11:30:39.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.971+0000 7fc553ded780 1 < canceled operation key=TQRZT37AB9WC tag=23YXZLVCC2GB type=write size=323769 2023-12-03T11:30:39.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.972+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=D76R2CQ7I535 type=del size=4096314 2023-12-03T11:30:39.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.974+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~IB766LI1CJOW.0 size=1048576 2023-12-03T11:30:39.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.975+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~IB766LI1CJOW.1 size=495535 2023-12-03T11:30:39.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.977+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=IB766LI1CJOW tag=J1C7EALDOA3E type=write size=2592687 2023-12-03T11:30:39.979 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.978+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=9SQ76HG08C20 type=write size=436960 2023-12-03T11:30:39.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.979+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=A45IT67QEVTT type=del 2023-12-03T11:30:39.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.980+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=2CYV38NO05JW type=del size=16751 2023-12-03T11:30:39.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.981+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~WM9YWYTPF3WO.0 size=1048576 2023-12-03T11:30:39.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.982+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~WM9YWYTPF3WO.1 size=1038806 2023-12-03T11:30:39.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.984+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=WM9YWYTPF3WO tag=NA0A7JXX89KZ type=write size=3135958 2023-12-03T11:30:39.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.986+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~JOPKTFGSCMUI.0 size=1016992 2023-12-03T11:30:39.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.986+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=JOPKTFGSCMUI tag=1MQL5X54VFUB type=write size=2065568 2023-12-03T11:30:39.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.988+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=Y4N0QBTSBL26 type=del 2023-12-03T11:30:39.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.989+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=DJ3GAOVOCV6H type=del size=533911 2023-12-03T11:30:39.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.991+0000 7fc553ded780 1 < suggested remove operation key=8GRA2MUBGWJI tag=50U0N49DJATK type=write 2023-12-03T11:30:39.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.993+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.0 size=1048576 2023-12-03T11:30:39.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.994+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.1 size=1048576 2023-12-03T11:30:39.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.995+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.2 size=826683 2023-12-03T11:30:39.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.995+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=YG25X1HR2W7Z tag=WL03IDQQRJ7Z type=write size=3972411 2023-12-03T11:30:39.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.996+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~7RJL2CQLGPUI.0 size=1048576 2023-12-03T11:30:39.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.997+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~7RJL2CQLGPUI.1 size=721109 2023-12-03T11:30:40.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:39.999+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=7RJL2CQLGPUI tag=4VU7038TDYGJ type=write size=2818261 2023-12-03T11:30:40.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.001+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=DXXYSDVGWVXI type=del size=2438355 2023-12-03T11:30:40.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.002+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=UH0ZPH0FNOIF type=write size=991332 2023-12-03T11:30:40.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.004+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~15CZCS9101YI.0 size=933828 2023-12-03T11:30:40.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.004+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=15CZCS9101YI tag=21B3R0UUHI7Z type=write size=1982404 2023-12-03T11:30:40.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.006+0000 7fc553ded780 1 < suggested update operation key=UKAL8FCVIRSH tag=M42KUYTVUBT7 type=del size=1572079 2023-12-03T11:30:40.009 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.009+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~Z8LDSOM6MXJC.0 size=1048576 2023-12-03T11:30:40.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.010+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~Z8LDSOM6MXJC.1 size=893601 2023-12-03T11:30:40.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.010+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=Z8LDSOM6MXJC tag=CAHQPZMDZYW9 type=write size=2990753 2023-12-03T11:30:40.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.011+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~YEJBKMJYIQXU.0 size=550495 2023-12-03T11:30:40.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.013+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=YEJBKMJYIQXU tag=TJBNSFXF949P type=write size=1599071 2023-12-03T11:30:40.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.015+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.0 size=1048576 2023-12-03T11:30:40.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.016+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.1 size=1048576 2023-12-03T11:30:40.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.017+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.2 size=1038124 2023-12-03T11:30:40.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.017+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=4SQNR4FUS0FD tag=JKQFBMKS2PNC type=write size=4183852 2023-12-03T11:30:40.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.018+0000 7fc553ded780 1 < completed write operation key=ZHK2M78A0SLQ tag=9PYR7912T71R type=write size=34798 2023-12-03T11:30:40.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.020+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~STE9JSCEWL90.0 size=1048576 2023-12-03T11:30:40.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.021+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~STE9JSCEWL90.1 size=260740 2023-12-03T11:30:40.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.021+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=STE9JSCEWL90 tag=TMTFF6521G0I type=write size=2357892 2023-12-03T11:30:40.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.023+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~24PMWUN78JJD.0 size=201755 2023-12-03T11:30:40.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.025+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=24PMWUN78JJD tag=HL8SOIZA79PN type=write size=1250331 2023-12-03T11:30:40.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.026+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=8NFUXAY87GLQ type=del size=2869327 2023-12-03T11:30:40.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.028+0000 7fc553ded780 1 < suggested remove operation key=ZUNOFY5N3J3H tag=XHSJ5Q9Q2KSC type=del 2023-12-03T11:30:40.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.032+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~JRY4PEVV7FSY.0 size=1048576 2023-12-03T11:30:40.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.033+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~JRY4PEVV7FSY.1 size=1023728 2023-12-03T11:30:40.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.033+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=JRY4PEVV7FSY tag=HBNR3F8R5MGR type=write size=3120880 2023-12-03T11:30:40.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.036+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=IKHA8JCZJMLU type=del 2023-12-03T11:30:40.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.038+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=R87A07NWPR2D type=del size=1165413 2023-12-03T11:30:40.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.039+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=330MRNFE7ZS1 type=del 2023-12-03T11:30:40.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.043+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Y45VFFLE7UQO.0 size=107819 2023-12-03T11:30:40.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.043+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=Y45VFFLE7UQO tag=WDV8VSQ5HPFU type=write size=1156395 2023-12-03T11:30:40.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.044+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.0 size=1048576 2023-12-03T11:30:40.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.045+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.1 size=1048576 2023-12-03T11:30:40.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.046+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~C1GUKXYMQN1Z.2 size=180651 2023-12-03T11:30:40.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.048+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=C1GUKXYMQN1Z tag=H0HKYED50T3A type=write size=3326379 2023-12-03T11:30:40.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.049+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=66U9BPTNK4C1 type=del size=2835794 2023-12-03T11:30:40.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.052+0000 7fc553ded780 1 < suggested update operation key=WDV4CW863V33 tag=D76R2CQ7I535 type=del size=4096314 2023-12-03T11:30:40.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.053+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=E170VJYEQRIO type=del size=3130800 2023-12-03T11:30:40.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.054+0000 7fc553ded780 1 < completed write operation key=TQRZT37AB9WC tag=9SQ76HG08C20 type=write size=436960 2023-12-03T11:30:40.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.057+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~SVYMIICOVAR9.0 size=1028492 2023-12-03T11:30:40.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.057+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=SVYMIICOVAR9 tag=4NHES3ZAMURS type=write size=2077068 2023-12-03T11:30:40.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.058+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=2CYV38NO05JW type=del 2023-12-03T11:30:40.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.059+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=40ASJ322WIMX type=del size=1092156 2023-12-03T11:30:40.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.060+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~JOPKTFGSCMUI.0 size=1016992 2023-12-03T11:30:40.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.062+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=JOPKTFGSCMUI tag=1MQL5X54VFUB type=write size=2065568 2023-12-03T11:30:40.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.064+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=N6JUL0NDPU0J type=del size=1210732 2023-12-03T11:30:40.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.065+0000 7fc553ded780 1 < canceled operation key=YE73HBF64CS7 tag=DJ3GAOVOCV6H type=del size=533911 2023-12-03T11:30:40.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.066+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=QMQJ38LGS64C type=write size=33237 2023-12-03T11:30:40.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.067+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.0 size=1048576 2023-12-03T11:30:40.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.068+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.1 size=1048576 2023-12-03T11:30:40.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.070+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~YG25X1HR2W7Z.2 size=826683 2023-12-03T11:30:40.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.072+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=YG25X1HR2W7Z tag=WL03IDQQRJ7Z type=write size=3972411 2023-12-03T11:30:40.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.074+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.0 size=1048576 2023-12-03T11:30:40.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.075+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.1 size=1048576 2023-12-03T11:30:40.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.076+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.2 size=112268 2023-12-03T11:30:40.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.076+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=L4QRMN1UKUDR tag=J374U8JSQKPH type=write size=3257996 2023-12-03T11:30:40.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.077+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=DXXYSDVGWVXI type=del 2023-12-03T11:30:40.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.078+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=BTW9B3XBLGWD type=del size=3317785 2023-12-03T11:30:40.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.079+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~15CZCS9101YI.0 size=933828 2023-12-03T11:30:40.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.081+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=15CZCS9101YI tag=21B3R0UUHI7Z type=write size=1982404 2023-12-03T11:30:40.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.082+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=AQM62B7KQ66G type=del size=2353876 2023-12-03T11:30:40.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.084+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~Z8LDSOM6MXJC.0 size=1048576 2023-12-03T11:30:40.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.084+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~Z8LDSOM6MXJC.1 size=893601 2023-12-03T11:30:40.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.087+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=Z8LDSOM6MXJC tag=CAHQPZMDZYW9 type=write size=2990753 2023-12-03T11:30:40.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.088+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=7FEAPTO9XDOO type=del size=3582661 2023-12-03T11:30:40.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.088+0000 7fc553ded780 1 < canceled part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.0 size=1048576 2023-12-03T11:30:40.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.088+0000 7fc553ded780 1 < canceled part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.1 size=1048576 2023-12-03T11:30:40.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.088+0000 7fc553ded780 1 < canceled part key=_multipart_CZP8OQRU641T.2~4SQNR4FUS0FD.2 size=1038124 2023-12-03T11:30:40.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.090+0000 7fc553ded780 1 < canceled multipart upload key=CZP8OQRU641T upload=4SQNR4FUS0FD tag=JKQFBMKS2PNC type=write size=4183852 2023-12-03T11:30:40.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.091+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=6EYDXS9A1B21 type=del size=3589293 2023-12-03T11:30:40.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.092+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~STE9JSCEWL90.0 size=1048576 2023-12-03T11:30:40.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.093+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~STE9JSCEWL90.1 size=260740 2023-12-03T11:30:40.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.096+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=STE9JSCEWL90 tag=TMTFF6521G0I type=write size=2357892 2023-12-03T11:30:40.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.098+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~EO0GWQVS0U7T.0 size=1048576 2023-12-03T11:30:40.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.099+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~EO0GWQVS0U7T.1 size=229678 2023-12-03T11:30:40.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.099+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=EO0GWQVS0U7T tag=HX32BT6GE7YQ type=write size=2326830 2023-12-03T11:30:40.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.100+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=8NFUXAY87GLQ type=del 2023-12-03T11:30:40.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.101+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=TTK5UXY4WV3Q type=write size=810669 2023-12-03T11:30:40.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.101+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~JRY4PEVV7FSY.0 size=1048576 2023-12-03T11:30:40.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.101+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~JRY4PEVV7FSY.1 size=1023728 2023-12-03T11:30:40.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.103+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=JRY4PEVV7FSY tag=HBNR3F8R5MGR type=write size=3120880 2023-12-03T11:30:40.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.105+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=NKVMAIVY1WOI type=del size=928654 2023-12-03T11:30:40.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.106+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=R87A07NWPR2D type=del 2023-12-03T11:30:40.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.107+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=21ZKWSHAKMHG type=del size=2121954 2023-12-03T11:30:40.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.108+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Y45VFFLE7UQO.0 size=107819 2023-12-03T11:30:40.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.110+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=Y45VFFLE7UQO tag=WDV8VSQ5HPFU type=write size=1156395 2023-12-03T11:30:40.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.111+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=9L2DCA6UOZKT type=del size=3556500 2023-12-03T11:30:40.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.113+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=66U9BPTNK4C1 type=del 2023-12-03T11:30:40.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.114+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=CNCHNAZWNSOL type=write size=672490 2023-12-03T11:30:40.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.115+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=E170VJYEQRIO type=del 2023-12-03T11:30:40.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.117+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.0 size=1048576 2023-12-03T11:30:40.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.118+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.1 size=1048576 2023-12-03T11:30:40.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.119+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.2 size=503171 2023-12-03T11:30:40.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.119+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=XUOYO9E0R8UP tag=9JYS8BSYP22O type=write size=3648899 2023-12-03T11:30:40.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.121+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~SVYMIICOVAR9.0 size=1028492 2023-12-03T11:30:40.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.123+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=SVYMIICOVAR9 tag=4NHES3ZAMURS type=write size=2077068 2023-12-03T11:30:40.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.124+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=M3FNJL34NZUK type=del size=4006603 2023-12-03T11:30:40.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.126+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=40ASJ322WIMX type=del 2023-12-03T11:30:40.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.127+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=9XXEC260DCSX type=write size=195194 2023-12-03T11:30:40.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.128+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=N6JUL0NDPU0J type=del 2023-12-03T11:30:40.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.130+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=8Z6QY4OK6X0G type=del size=565335 2023-12-03T11:30:40.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.131+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=QMQJ38LGS64C type=write size=33237 2023-12-03T11:30:40.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.133+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~7OYEZ0MY6SJ4.0 size=447131 2023-12-03T11:30:40.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.133+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=7OYEZ0MY6SJ4 tag=341J2L4J58B9 type=write size=1495707 2023-12-03T11:30:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.135+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.0 size=1048576 2023-12-03T11:30:40.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.135+0000 7fc553ded780 1 < canceled part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.1 size=1048576 2023-12-03T11:30:40.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.136+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~L4QRMN1UKUDR.2 size=112268 2023-12-03T11:30:40.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.138+0000 7fc553ded780 1 < canceled multipart upload key=CZP8OQRU641T upload=L4QRMN1UKUDR tag=J374U8JSQKPH type=write size=3257996 2023-12-03T11:30:40.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.139+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=Z0ZVW1RQCF2U type=del size=2925653 2023-12-03T11:30:40.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.141+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=BTW9B3XBLGWD type=del 2023-12-03T11:30:40.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.143+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.0 size=1048576 2023-12-03T11:30:40.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.144+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.1 size=1048576 2023-12-03T11:30:40.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.145+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.2 size=672420 2023-12-03T11:30:40.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.145+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=KK2OCD53APYI tag=DFUC4LIENNDJ type=write size=3818148 2023-12-03T11:30:40.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.146+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=AQM62B7KQ66G type=del 2023-12-03T11:30:40.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.148+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~3BX126ZB0GT0.0 size=977050 2023-12-03T11:30:40.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.148+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=3BX126ZB0GT0 tag=6IK6PYWK7IC4 type=write size=2025626 2023-12-03T11:30:40.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.150+0000 7fc553ded780 1 < canceled operation key=8GRA2MUBGWJI tag=7FEAPTO9XDOO type=del size=3582661 2023-12-03T11:30:40.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.151+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=6UK8ZMQBNOBC type=del size=2405272 2023-12-03T11:30:40.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.152+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=6EYDXS9A1B21 type=del 2023-12-03T11:30:40.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.155+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.0 size=1048576 2023-12-03T11:30:40.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.156+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.1 size=1048576 2023-12-03T11:30:40.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.159+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.2 size=30760 2023-12-03T11:30:40.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.159+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=JPUJYCMNOA99 tag=XX7XTUU2SX1G type=write size=3176488 2023-12-03T11:30:40.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.160+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~EO0GWQVS0U7T.0 size=1048576 2023-12-03T11:30:40.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.161+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~EO0GWQVS0U7T.1 size=229678 2023-12-03T11:30:40.164 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.163+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=EO0GWQVS0U7T tag=HX32BT6GE7YQ type=write size=2326830 2023-12-03T11:30:40.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.164+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=Z5BXB27QUAZ1 type=del size=2959946 2023-12-03T11:30:40.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.166+0000 7fc553ded780 1 < completed write operation key=X741JIRT7RIG tag=TTK5UXY4WV3Q type=write size=810669 2023-12-03T11:30:40.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.167+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=O6PIVD6R28HP type=del size=4016846 2023-12-03T11:30:40.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.169+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=NKVMAIVY1WOI type=del 2023-12-03T11:30:40.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.170+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=4NMBM4RE5WWX type=del size=768806 2023-12-03T11:30:40.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.172+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=21ZKWSHAKMHG type=del 2023-12-03T11:30:40.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.173+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=BL3ON43WIXUG type=del size=912788 2023-12-03T11:30:40.176 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.175+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=9L2DCA6UOZKT type=del 2023-12-03T11:30:40.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.176+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=HSPSC3LQ5FR1 type=del size=1781724 2023-12-03T11:30:40.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.177+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=CNCHNAZWNSOL type=write size=672490 2023-12-03T11:30:40.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.179+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=GF2AWG1YL5FT type=del size=730057 2023-12-03T11:30:40.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.180+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.0 size=1048576 2023-12-03T11:30:40.182 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.181+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.1 size=1048576 2023-12-03T11:30:40.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.182+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~XUOYO9E0R8UP.2 size=503171 2023-12-03T11:30:40.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.184+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=XUOYO9E0R8UP tag=9JYS8BSYP22O type=write size=3648899 2023-12-03T11:30:40.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.187+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.0 size=1048576 2023-12-03T11:30:40.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.188+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.1 size=1048576 2023-12-03T11:30:40.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.189+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.2 size=31130 2023-12-03T11:30:40.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.189+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=YSQNM4N2A7JK tag=ON3L0TK7S6E4 type=write size=3176858 2023-12-03T11:30:40.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.191+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=M3FNJL34NZUK type=del size=4006603 2023-12-03T11:30:40.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.192+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=GABI9JTYW8EC type=del size=3282333 2023-12-03T11:30:40.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.194+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=9XXEC260DCSX type=write size=195194 2023-12-03T11:30:40.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.195+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=E7FZ1FA1ZUN9 type=write size=803891 2023-12-03T11:30:40.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.196+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=8Z6QY4OK6X0G type=del 2023-12-03T11:30:40.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.197+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=DYO46R1QBDZS type=del size=3862870 2023-12-03T11:30:40.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.199+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~7OYEZ0MY6SJ4.0 size=447131 2023-12-03T11:30:40.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.201+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=7OYEZ0MY6SJ4 tag=341J2L4J58B9 type=write size=1495707 2023-12-03T11:30:40.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.202+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=4QDUNYO5FTK9 type=del size=1418513 2023-12-03T11:30:40.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.204+0000 7fc553ded780 1 < canceled operation key=8GRA2MUBGWJI tag=Z0ZVW1RQCF2U type=del size=2925653 2023-12-03T11:30:40.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.205+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=YIHNQPLWOUNS type=del size=1176814 2023-12-03T11:30:40.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.206+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.0 size=1048576 2023-12-03T11:30:40.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.206+0000 7fc553ded780 1 < canceled part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.1 size=1048576 2023-12-03T11:30:40.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.207+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~KK2OCD53APYI.2 size=672420 2023-12-03T11:30:40.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.209+0000 7fc553ded780 1 < canceled multipart upload key=J664UUE697Q8 upload=KK2OCD53APYI tag=DFUC4LIENNDJ type=write size=3818148 2023-12-03T11:30:40.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.211+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=MH3Q2P8TYP1V type=del size=689013 2023-12-03T11:30:40.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.212+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~3BX126ZB0GT0.0 size=977050 2023-12-03T11:30:40.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.214+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=3BX126ZB0GT0 tag=6IK6PYWK7IC4 type=write size=2025626 2023-12-03T11:30:40.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.215+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=SG3YX4M7UH23 type=del size=2748659 2023-12-03T11:30:40.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.216+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=6UK8ZMQBNOBC type=del 2023-12-03T11:30:40.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.218+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=G4KMFNB5OCIW type=del size=3988541 2023-12-03T11:30:40.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.219+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.0 size=1048576 2023-12-03T11:30:40.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.220+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.1 size=1048576 2023-12-03T11:30:40.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.221+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~JPUJYCMNOA99.2 size=30760 2023-12-03T11:30:40.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.223+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=JPUJYCMNOA99 tag=XX7XTUU2SX1G type=write size=3176488 2023-12-03T11:30:40.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.224+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=XYOC1I3Q5OJJ type=del size=1474870 2023-12-03T11:30:40.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.226+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=Z5BXB27QUAZ1 type=del 2023-12-03T11:30:40.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.228+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.0 size=1048576 2023-12-03T11:30:40.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.229+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.1 size=1048576 2023-12-03T11:30:40.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.230+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.2 size=448216 2023-12-03T11:30:40.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.230+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=IJ6SWVLUKWHT tag=B0X4F3DHBXJK type=write size=3593944 2023-12-03T11:30:40.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.232+0000 7fc553ded780 1 < suggested update operation key=ZUNOFY5N3J3H tag=O6PIVD6R28HP type=del size=4016846 2023-12-03T11:30:40.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.234+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=W35S6T5OG77N type=del size=2347048 2023-12-03T11:30:40.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.235+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=4NMBM4RE5WWX type=del 2023-12-03T11:30:40.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.236+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=0OKXX7BJ8NG6 type=write size=97772 2023-12-03T11:30:40.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.237+0000 7fc553ded780 1 < canceled operation key=J664UUE697Q8 tag=BL3ON43WIXUG type=del size=912788 2023-12-03T11:30:40.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.239+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=3CSYLGJFROOJ type=del size=1044874 2023-12-03T11:30:40.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.240+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=HSPSC3LQ5FR1 type=del 2023-12-03T11:30:40.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.242+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~UWKFTIDODFTC.0 size=683747 2023-12-03T11:30:40.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.242+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=UWKFTIDODFTC tag=L48U0NARPC83 type=write size=1732323 2023-12-03T11:30:40.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.244+0000 7fc553ded780 1 < canceled operation key=0H5OZMBMLQFJ tag=GF2AWG1YL5FT type=del size=730057 2023-12-03T11:30:40.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.246+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.0 size=1048576 2023-12-03T11:30:40.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.249+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.1 size=1048576 2023-12-03T11:30:40.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.250+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.2 size=346056 2023-12-03T11:30:40.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.250+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=0NWAYRJYBCII tag=3YSY0Y7S6N5L type=write size=3491784 2023-12-03T11:30:40.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.251+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.0 size=1048576 2023-12-03T11:30:40.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.251+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.1 size=1048576 2023-12-03T11:30:40.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.251+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~YSQNM4N2A7JK.2 size=31130 2023-12-03T11:30:40.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.253+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=YSQNM4N2A7JK tag=ON3L0TK7S6E4 type=write size=3176858 2023-12-03T11:30:40.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.255+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~P3VT3L5V814Y.0 size=69510 2023-12-03T11:30:40.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.255+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=P3VT3L5V814Y tag=NC17KOO7Y5ID type=write size=1118086 2023-12-03T11:30:40.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.256+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=GABI9JTYW8EC type=del 2023-12-03T11:30:40.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.257+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=AHTIQ9VMF6P7 type=del size=4054533 2023-12-03T11:30:40.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.260+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=E7FZ1FA1ZUN9 type=write size=803891 2023-12-03T11:30:40.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.262+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~CI2GKX4G115O.0 size=225934 2023-12-03T11:30:40.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.262+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=CI2GKX4G115O tag=4Q5E47709HAT type=write size=1274510 2023-12-03T11:30:40.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.264+0000 7fc553ded780 1 < suggested update operation key=WOKG1DISIKK0 tag=DYO46R1QBDZS type=del size=3862870 2023-12-03T11:30:40.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.266+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=EPOSAOLCK64R type=del size=494305 2023-12-03T11:30:40.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.267+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=4QDUNYO5FTK9 type=del 2023-12-03T11:30:40.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.271+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~MCZWNFDDCFVQ.0 size=1048576 2023-12-03T11:30:40.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.272+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~MCZWNFDDCFVQ.1 size=880314 2023-12-03T11:30:40.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.272+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=MCZWNFDDCFVQ tag=158QDLTP0EXF type=write size=2977466 2023-12-03T11:30:40.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.273+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=YIHNQPLWOUNS type=del 2023-12-03T11:30:40.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.274+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=GVD9ZBRNNVQ1 type=del size=1859489 2023-12-03T11:30:40.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.276+0000 7fc553ded780 -1 < failed to complete operation key=L4ME9RYRR4NG tag=MH3Q2P8TYP1V type=del size=689013: (22) Invalid argument 2023-12-03T11:30:40.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.277+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=J73JFWYF8B13 type=del size=1810235 2023-12-03T11:30:40.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.278+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=SG3YX4M7UH23 type=del 2023-12-03T11:30:40.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.280+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=4ML2ZZEIS1AP type=write size=598687 2023-12-03T11:30:40.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.282+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=G4KMFNB5OCIW type=del size=3988541: (22) Invalid argument 2023-12-03T11:30:40.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.284+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~93TV996FX2UT.0 size=1048576 2023-12-03T11:30:40.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.285+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~93TV996FX2UT.1 size=893389 2023-12-03T11:30:40.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.285+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=93TV996FX2UT tag=Z3FIB747IV66 type=write size=2990541 2023-12-03T11:30:40.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.286+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=XYOC1I3Q5OJJ type=del 2023-12-03T11:30:40.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.287+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=1A1E0WHBHCN9 type=write size=1034501 2023-12-03T11:30:40.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.289+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.0 size=1048576 2023-12-03T11:30:40.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.290+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.1 size=1048576 2023-12-03T11:30:40.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.290+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IJ6SWVLUKWHT.2 size=448216 2023-12-03T11:30:40.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.292+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=IJ6SWVLUKWHT tag=B0X4F3DHBXJK type=write size=3593944 2023-12-03T11:30:40.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.294+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=9XZZTVJKRIH8 type=del size=3455091 2023-12-03T11:30:40.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.295+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=W35S6T5OG77N type=del 2023-12-03T11:30:40.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.297+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~49OYRXEKLNXH.0 size=759312 2023-12-03T11:30:40.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.297+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=49OYRXEKLNXH tag=BGTZ4GAV5H99 type=write size=1807888 2023-12-03T11:30:40.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.300+0000 7fc553ded780 1 < suggested remove operation key=L4ME9RYRR4NG tag=0OKXX7BJ8NG6 type=write 2023-12-03T11:30:40.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.302+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~PNYUOHWCHMUO.0 size=374792 2023-12-03T11:30:40.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.302+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=PNYUOHWCHMUO tag=3I6V2ERZ4EXX type=write size=1423368 2023-12-03T11:30:40.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.303+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=3CSYLGJFROOJ type=del 2023-12-03T11:30:40.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.305+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=0Z2IATZUX4KU type=write size=322428 2023-12-03T11:30:40.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.306+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~UWKFTIDODFTC.0 size=683747 2023-12-03T11:30:40.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.308+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=UWKFTIDODFTC tag=L48U0NARPC83 type=write size=1732323 2023-12-03T11:30:40.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.309+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=3A69MDUJV0MR type=write size=29770 2023-12-03T11:30:40.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.309+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.0 size=1048576 2023-12-03T11:30:40.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.310+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.1 size=1048576 2023-12-03T11:30:40.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.311+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~0NWAYRJYBCII.2 size=346056 2023-12-03T11:30:40.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.313+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=0NWAYRJYBCII tag=3YSY0Y7S6N5L type=write size=3491784 2023-12-03T11:30:40.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.316+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~3HWMJSV27PZ6.0 size=1048576 2023-12-03T11:30:40.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.318+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~3HWMJSV27PZ6.1 size=168410 2023-12-03T11:30:40.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.318+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=3HWMJSV27PZ6 tag=MDMJFX91BTCB type=write size=2265562 2023-12-03T11:30:40.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.319+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~P3VT3L5V814Y.0 size=69510 2023-12-03T11:30:40.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.321+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=P3VT3L5V814Y tag=NC17KOO7Y5ID type=write size=1118086 2023-12-03T11:30:40.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.322+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=IU7PJQFBCHFB type=del size=3405525 2023-12-03T11:30:40.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.324+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=AHTIQ9VMF6P7 type=del 2023-12-03T11:30:40.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.326+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~4C990BKRI4RB.0 size=1048576 2023-12-03T11:30:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.327+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~4C990BKRI4RB.1 size=811355 2023-12-03T11:30:40.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.327+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=4C990BKRI4RB tag=1XZE6X17GMHV type=write size=2908507 2023-12-03T11:30:40.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.328+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~CI2GKX4G115O.0 size=225934 2023-12-03T11:30:40.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.330+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=CI2GKX4G115O tag=4Q5E47709HAT type=write size=1274510 2023-12-03T11:30:40.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.333+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.0 size=1048576 2023-12-03T11:30:40.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.334+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.1 size=1048576 2023-12-03T11:30:40.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.335+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.2 size=32570 2023-12-03T11:30:40.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.335+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=T04N5EZIUFOS tag=OW1V9JDQYG2G type=write size=3178298 2023-12-03T11:30:40.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.336+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=EPOSAOLCK64R type=del 2023-12-03T11:30:40.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.338+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~AO68F537SO6T.0 size=18544 2023-12-03T11:30:40.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.338+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=AO68F537SO6T tag=ER5QDVEO8CDP type=write size=1067120 2023-12-03T11:30:40.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.339+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~MCZWNFDDCFVQ.0 size=1048576 2023-12-03T11:30:40.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.340+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~MCZWNFDDCFVQ.1 size=880314 2023-12-03T11:30:40.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.342+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=MCZWNFDDCFVQ tag=158QDLTP0EXF type=write size=2977466 2023-12-03T11:30:40.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.344+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=QFZWYQP3ZSGD type=del size=778816 2023-12-03T11:30:40.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.345+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=GVD9ZBRNNVQ1 type=del 2023-12-03T11:30:40.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.346+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=GIQILJ30YC82 type=del size=3297202 2023-12-03T11:30:40.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.348+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=J73JFWYF8B13 type=del 2023-12-03T11:30:40.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.350+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.0 size=1048576 2023-12-03T11:30:40.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.351+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.1 size=1048576 2023-12-03T11:30:40.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.352+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.2 size=455948 2023-12-03T11:30:40.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.352+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=VE1B4KO8EVZ6 tag=LXQF6KPTIL75 type=write size=3601676 2023-12-03T11:30:40.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.353+0000 7fc553ded780 1 < canceled operation key=WOKG1DISIKK0 tag=4ML2ZZEIS1AP type=write size=598687 2023-12-03T11:30:40.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.355+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=U3H1ZJDRA1Q1 type=del size=1854159 2023-12-03T11:30:40.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.356+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~93TV996FX2UT.0 size=1048576 2023-12-03T11:30:40.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.356+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~93TV996FX2UT.1 size=893389 2023-12-03T11:30:40.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.359+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=93TV996FX2UT tag=Z3FIB747IV66 type=write size=2990541 2023-12-03T11:30:40.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.360+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=OSLTLG5KJ5DR type=del size=2356776 2023-12-03T11:30:40.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.361+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=1A1E0WHBHCN9 type=write size=1034501 2023-12-03T11:30:40.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.363+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~GE20DX2FGP92.0 size=1048576 2023-12-03T11:30:40.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.364+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~GE20DX2FGP92.1 size=721862 2023-12-03T11:30:40.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.364+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=GE20DX2FGP92 tag=XXTZLHSYJCPO type=write size=2819014 2023-12-03T11:30:40.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.366+0000 7fc553ded780 1 < canceled operation key=UKAL8FCVIRSH tag=9XZZTVJKRIH8 type=del size=3455091 2023-12-03T11:30:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.368+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~WEZI826F6QE7.0 size=949247 2023-12-03T11:30:40.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.368+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=WEZI826F6QE7 tag=XOJI75V7PI76 type=write size=1997823 2023-12-03T11:30:40.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.369+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~49OYRXEKLNXH.0 size=759312 2023-12-03T11:30:40.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.371+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=49OYRXEKLNXH tag=BGTZ4GAV5H99 type=write size=1807888 2023-12-03T11:30:40.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.373+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=I2VZP6UKIHD8 type=del size=2769633 2023-12-03T11:30:40.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.374+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~PNYUOHWCHMUO.0 size=374792 2023-12-03T11:30:40.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.376+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=PNYUOHWCHMUO tag=3I6V2ERZ4EXX type=write size=1423368 2023-12-03T11:30:40.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.377+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=F3H9NE45DS0W type=del size=3670596 2023-12-03T11:30:40.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.378+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=0Z2IATZUX4KU type=write size=322428 2023-12-03T11:30:40.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.379+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=MHIWRPK33S0D type=del size=3430480 2023-12-03T11:30:40.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.381+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=3A69MDUJV0MR type=write size=29770 2023-12-03T11:30:40.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.382+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=C6JC7GMNM502 type=del size=3161712 2023-12-03T11:30:40.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.383+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~3HWMJSV27PZ6.0 size=1048576 2023-12-03T11:30:40.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.384+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~3HWMJSV27PZ6.1 size=168410 2023-12-03T11:30:40.387 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.386+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=3HWMJSV27PZ6 tag=MDMJFX91BTCB type=write size=2265562 2023-12-03T11:30:40.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.387+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=LKJF4U6I04CX type=del size=1881372 2023-12-03T11:30:40.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.389+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=IU7PJQFBCHFB type=del size=3405525 2023-12-03T11:30:40.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.391+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=Q2OZ3WGK7SRF type=write size=527654 2023-12-03T11:30:40.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.392+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~4C990BKRI4RB.0 size=1048576 2023-12-03T11:30:40.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.393+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~4C990BKRI4RB.1 size=811355 2023-12-03T11:30:40.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.395+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=4C990BKRI4RB tag=1XZE6X17GMHV type=write size=2908507 2023-12-03T11:30:40.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.398+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.0 size=1048576 2023-12-03T11:30:40.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.401+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.1 size=1048576 2023-12-03T11:30:40.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.402+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.2 size=262762 2023-12-03T11:30:40.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.402+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=IEMYSX82QGBZ tag=OV379GZHCMO4 type=write size=3408490 2023-12-03T11:30:40.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.403+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.0 size=1048576 2023-12-03T11:30:40.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.404+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.1 size=1048576 2023-12-03T11:30:40.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.404+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~T04N5EZIUFOS.2 size=32570 2023-12-03T11:30:40.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.406+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=T04N5EZIUFOS tag=OW1V9JDQYG2G type=write size=3178298 2023-12-03T11:30:40.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.407+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=LB1Y2YFU4KZ8 type=del size=3570427 2023-12-03T11:30:40.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.408+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~AO68F537SO6T.0 size=18544 2023-12-03T11:30:40.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.410+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=AO68F537SO6T tag=ER5QDVEO8CDP type=write size=1067120 2023-12-03T11:30:40.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.412+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=K6RXOAOHL412 type=write size=815588 2023-12-03T11:30:40.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.413+0000 7fc553ded780 -1 < failed to complete operation key=0H5OZMBMLQFJ tag=QFZWYQP3ZSGD type=del size=778816: (22) Invalid argument 2023-12-03T11:30:40.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.415+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~U513F6GOP09S.0 size=1048576 2023-12-03T11:30:40.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.416+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~U513F6GOP09S.1 size=112234 2023-12-03T11:30:40.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.416+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=U513F6GOP09S tag=0YXNUMIAT3GA type=write size=2209386 2023-12-03T11:30:40.419 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.418+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=GIQILJ30YC82 type=del 2023-12-03T11:30:40.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.420+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.0 size=1048576 2023-12-03T11:30:40.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.421+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.1 size=1048576 2023-12-03T11:30:40.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.422+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.2 size=1002028 2023-12-03T11:30:40.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.422+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=PJT0QWPMUSFD tag=PD1LB6H4PQM6 type=write size=4147756 2023-12-03T11:30:40.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.422+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.0 size=1048576 2023-12-03T11:30:40.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.423+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.1 size=1048576 2023-12-03T11:30:40.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.424+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~VE1B4KO8EVZ6.2 size=455948 2023-12-03T11:30:40.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.426+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=VE1B4KO8EVZ6 tag=LXQF6KPTIL75 type=write size=3601676 2023-12-03T11:30:40.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.428+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~Q4SUTGGV3SIW.0 size=1048576 2023-12-03T11:30:40.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.429+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~Q4SUTGGV3SIW.1 size=471046 2023-12-03T11:30:40.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.429+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=Q4SUTGGV3SIW tag=BBYLL3JBGNC5 type=write size=2568198 2023-12-03T11:30:40.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.431+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=U3H1ZJDRA1Q1 type=del 2023-12-03T11:30:40.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.433+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~84923V5YX96J.0 size=1048576 2023-12-03T11:30:40.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.434+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~84923V5YX96J.1 size=787268 2023-12-03T11:30:40.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.434+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=84923V5YX96J tag=2NW158J6LG4I type=write size=2884420 2023-12-03T11:30:40.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.436+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=OSLTLG5KJ5DR type=del 2023-12-03T11:30:40.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.437+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=6UOIPFOC5JL8 type=del size=3879331 2023-12-03T11:30:40.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.438+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~GE20DX2FGP92.0 size=1048576 2023-12-03T11:30:40.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.439+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~GE20DX2FGP92.1 size=721862 2023-12-03T11:30:40.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.441+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=GE20DX2FGP92 tag=XXTZLHSYJCPO type=write size=2819014 2023-12-03T11:30:40.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.443+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=3SSJM2UEBZLF type=del size=3573775 2023-12-03T11:30:40.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.444+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~WEZI826F6QE7.0 size=949247 2023-12-03T11:30:40.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.446+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=WEZI826F6QE7 tag=XOJI75V7PI76 type=write size=1997823 2023-12-03T11:30:40.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.448+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~ZXHQ0XETRSE2.0 size=1048576 2023-12-03T11:30:40.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.449+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~ZXHQ0XETRSE2.1 size=796858 2023-12-03T11:30:40.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.449+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=ZXHQ0XETRSE2 tag=F0F5AA9L1S8O type=write size=2894010 2023-12-03T11:30:40.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.450+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=I2VZP6UKIHD8 type=del 2023-12-03T11:30:40.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.452+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=JVIP77ZBN9XK type=write size=26470 2023-12-03T11:30:40.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.453+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=F3H9NE45DS0W type=del 2023-12-03T11:30:40.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.454+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=7Z4CXO2MFK39 type=del size=1517951 2023-12-03T11:30:40.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.455+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=MHIWRPK33S0D type=del 2023-12-03T11:30:40.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.456+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=YY2ECJI52BZ3 type=del size=3715155 2023-12-03T11:30:40.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.458+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=C6JC7GMNM502 type=del 2023-12-03T11:30:40.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.460+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~3Q6T1QZTEQTJ.0 size=926248 2023-12-03T11:30:40.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.460+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=3Q6T1QZTEQTJ tag=YBEU22K1R18S type=write size=1974824 2023-12-03T11:30:40.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.461+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=LKJF4U6I04CX type=del 2023-12-03T11:30:40.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.462+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=XD5Q5Z86H1I2 type=del size=3216845 2023-12-03T11:30:40.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.464+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=Q2OZ3WGK7SRF type=write size=527654 2023-12-03T11:30:40.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.465+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=93US3IJS20XH type=write size=1023849 2023-12-03T11:30:40.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.466+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.0 size=1048576 2023-12-03T11:30:40.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.467+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.1 size=1048576 2023-12-03T11:30:40.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.468+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~IEMYSX82QGBZ.2 size=262762 2023-12-03T11:30:40.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.470+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=IEMYSX82QGBZ tag=OV379GZHCMO4 type=write size=3408490 2023-12-03T11:30:40.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.472+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JR1O48OXGUR8.0 size=1048576 2023-12-03T11:30:40.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.473+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JR1O48OXGUR8.1 size=34999 2023-12-03T11:30:40.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.473+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=JR1O48OXGUR8 tag=6UYFQHQKMTZG type=write size=2132151 2023-12-03T11:30:40.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.474+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=LB1Y2YFU4KZ8 type=del size=3570427 2023-12-03T11:30:40.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.476+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=PVXOMSYAXFBZ type=del size=1758990 2023-12-03T11:30:40.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.477+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=K6RXOAOHL412 type=write size=815588 2023-12-03T11:30:40.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.478+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=EH0XIULUUKTI type=del size=2232687 2023-12-03T11:30:40.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.479+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~U513F6GOP09S.0 size=1048576 2023-12-03T11:30:40.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.481+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~U513F6GOP09S.1 size=112234 2023-12-03T11:30:40.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.483+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=U513F6GOP09S tag=0YXNUMIAT3GA type=write size=2209386 2023-12-03T11:30:40.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.485+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~ME1GNC1DY9YX.0 size=590109 2023-12-03T11:30:40.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.485+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=ME1GNC1DY9YX tag=P24BA51FRLFG type=write size=1638685 2023-12-03T11:30:40.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.486+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.0 size=1048576 2023-12-03T11:30:40.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.487+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.1 size=1048576 2023-12-03T11:30:40.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.488+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~PJT0QWPMUSFD.2 size=1002028 2023-12-03T11:30:40.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.492+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=PJT0QWPMUSFD tag=PD1LB6H4PQM6 type=write size=4147756 2023-12-03T11:30:40.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.494+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=ZM10MNQQEJC3 type=write size=381214 2023-12-03T11:30:40.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.495+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~Q4SUTGGV3SIW.0 size=1048576 2023-12-03T11:30:40.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.496+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~Q4SUTGGV3SIW.1 size=471046 2023-12-03T11:30:40.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.498+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=Q4SUTGGV3SIW tag=BBYLL3JBGNC5 type=write size=2568198 2023-12-03T11:30:40.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.499+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=VJGN00KADQGL type=del size=2681210 2023-12-03T11:30:40.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.501+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~84923V5YX96J.0 size=1048576 2023-12-03T11:30:40.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.501+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~84923V5YX96J.1 size=787268 2023-12-03T11:30:40.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.503+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=84923V5YX96J tag=2NW158J6LG4I type=write size=2884420 2023-12-03T11:30:40.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.504+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=1ZPXRJW0ILX1 type=del size=2549143 2023-12-03T11:30:40.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.506+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=6UOIPFOC5JL8 type=del 2023-12-03T11:30:40.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.507+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=2FF75T25B544 type=del size=3687677 2023-12-03T11:30:40.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.508+0000 7fc553ded780 1 < canceled operation key=FC4WU6SMNEWV tag=3SSJM2UEBZLF type=del size=3573775 2023-12-03T11:30:40.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.509+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=2DBHP7GX17VU type=del size=2612167 2023-12-03T11:30:40.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.510+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~ZXHQ0XETRSE2.0 size=1048576 2023-12-03T11:30:40.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.511+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~ZXHQ0XETRSE2.1 size=796858 2023-12-03T11:30:40.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.513+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=ZXHQ0XETRSE2 tag=F0F5AA9L1S8O type=write size=2894010 2023-12-03T11:30:40.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.515+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=PDQMA514X9YJ type=write size=166424 2023-12-03T11:30:40.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.516+0000 7fc553ded780 1 < completed write operation key=6OW429S51UWN tag=JVIP77ZBN9XK type=write size=26470 2023-12-03T11:30:40.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.518+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.0 size=1048576 2023-12-03T11:30:40.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.519+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.1 size=1048576 2023-12-03T11:30:40.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.520+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.2 size=1000849 2023-12-03T11:30:40.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.520+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=V94U6X6RBM5S tag=HAB5WFIX6X4R type=write size=4146577 2023-12-03T11:30:40.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.522+0000 7fc553ded780 1 < suggested remove operation key=YE73HBF64CS7 tag=7Z4CXO2MFK39 type=del 2023-12-03T11:30:40.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.523+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=Y8ZX1LILC4AJ type=write size=380561 2023-12-03T11:30:40.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.524+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=YY2ECJI52BZ3 type=del 2023-12-03T11:30:40.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.526+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~5UPABRCS3ZVR.0 size=405261 2023-12-03T11:30:40.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.526+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=5UPABRCS3ZVR tag=JWFBEY74CTX2 type=write size=1453837 2023-12-03T11:30:40.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.528+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~3Q6T1QZTEQTJ.0 size=926248 2023-12-03T11:30:40.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.530+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=3Q6T1QZTEQTJ tag=YBEU22K1R18S type=write size=1974824 2023-12-03T11:30:40.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.533+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~KAMBT96R3ZBX.0 size=777378 2023-12-03T11:30:40.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.533+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=KAMBT96R3ZBX tag=QSYF5ZTJLVAK type=write size=1825954 2023-12-03T11:30:40.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.534+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=XD5Q5Z86H1I2 type=del 2023-12-03T11:30:40.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.536+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.0 size=1048576 2023-12-03T11:30:40.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.537+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.1 size=1048576 2023-12-03T11:30:40.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.538+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.2 size=654025 2023-12-03T11:30:40.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.538+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=J80KR78KPJS3 tag=YZHW3HLSM86P type=write size=3799753 2023-12-03T11:30:40.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.540+0000 7fc553ded780 1 < suggested remove operation key=6OW429S51UWN tag=93US3IJS20XH type=write 2023-12-03T11:30:40.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.542+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.0 size=1048576 2023-12-03T11:30:40.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.543+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.1 size=1048576 2023-12-03T11:30:40.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.544+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.2 size=593320 2023-12-03T11:30:40.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.544+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=TE0571S0WHN5 tag=E0CZ8N73N9ZW type=write size=3739048 2023-12-03T11:30:40.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.545+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JR1O48OXGUR8.0 size=1048576 2023-12-03T11:30:40.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.546+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JR1O48OXGUR8.1 size=34999 2023-12-03T11:30:40.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.548+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=JR1O48OXGUR8 tag=6UYFQHQKMTZG type=write size=2132151 2023-12-03T11:30:40.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.549+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=KG1YQ4I7CGZY type=del size=2895735 2023-12-03T11:30:40.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.551+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=PVXOMSYAXFBZ type=del 2023-12-03T11:30:40.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.552+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=NRMLV12CBC1J type=del size=3224410 2023-12-03T11:30:40.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.554+0000 7fc553ded780 1 < suggested remove operation key=SWTRHXTAR0A1 tag=EH0XIULUUKTI type=del 2023-12-03T11:30:40.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.557+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~LRVX0Z2HORUY.0 size=334241 2023-12-03T11:30:40.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.557+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=LRVX0Z2HORUY tag=BH7OUE0U3125 type=write size=1382817 2023-12-03T11:30:40.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.558+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~ME1GNC1DY9YX.0 size=590109 2023-12-03T11:30:40.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.560+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=ME1GNC1DY9YX tag=P24BA51FRLFG type=write size=1638685 2023-12-03T11:30:40.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.562+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~SJ981LPDQ9P2.0 size=1048576 2023-12-03T11:30:40.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.563+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~SJ981LPDQ9P2.1 size=650107 2023-12-03T11:30:40.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.563+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=SJ981LPDQ9P2 tag=8RBM4TSVC13I type=write size=2747259 2023-12-03T11:30:40.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.564+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=ZM10MNQQEJC3 type=write size=381214 2023-12-03T11:30:40.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.566+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.0 size=1048576 2023-12-03T11:30:40.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.567+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.1 size=1048576 2023-12-03T11:30:40.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.568+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.2 size=544361 2023-12-03T11:30:40.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.568+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=B792IG9Y78N5 tag=URTDB5OOBJFL type=write size=3690089 2023-12-03T11:30:40.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.569+0000 7fc553ded780 -1 < failed to complete operation key=6OW429S51UWN tag=VJGN00KADQGL type=del size=2681210: (22) Invalid argument 2023-12-03T11:30:40.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.571+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=E8709FKOH6M2 type=del size=2616424 2023-12-03T11:30:40.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.572+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=1ZPXRJW0ILX1 type=del 2023-12-03T11:30:40.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.573+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=VJOWGOTIWXFJ type=write size=862675 2023-12-03T11:30:40.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.574+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=2FF75T25B544 type=del size=3687677 2023-12-03T11:30:40.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.576+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=2W43PVY0Q0DJ type=del size=2393316 2023-12-03T11:30:40.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.577+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=2DBHP7GX17VU type=del 2023-12-03T11:30:40.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.579+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.0 size=1048576 2023-12-03T11:30:40.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.580+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.1 size=1048576 2023-12-03T11:30:40.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.581+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.2 size=432154 2023-12-03T11:30:40.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.581+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=JLZ020W1HTG4 tag=YRNDWEYXJY88 type=write size=3577882 2023-12-03T11:30:40.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.582+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=PDQMA514X9YJ type=write size=166424 2023-12-03T11:30:40.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.584+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=IKFSIHM1GL1U type=del size=3473303 2023-12-03T11:30:40.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.585+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.0 size=1048576 2023-12-03T11:30:40.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.585+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.1 size=1048576 2023-12-03T11:30:40.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.586+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~V94U6X6RBM5S.2 size=1000849 2023-12-03T11:30:40.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.588+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=V94U6X6RBM5S tag=HAB5WFIX6X4R type=write size=4146577 2023-12-03T11:30:40.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.589+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=LA5J3FBTR0F5 type=write size=231149 2023-12-03T11:30:40.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.590+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=Y8ZX1LILC4AJ type=write size=380561 2023-12-03T11:30:40.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.592+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~0OH5EB8GKPQ6.0 size=915973 2023-12-03T11:30:40.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.592+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=0OH5EB8GKPQ6 tag=MOEDPWPQYVWP type=write size=1964549 2023-12-03T11:30:40.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.594+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~5UPABRCS3ZVR.0 size=405261 2023-12-03T11:30:40.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.596+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=5UPABRCS3ZVR tag=JWFBEY74CTX2 type=write size=1453837 2023-12-03T11:30:40.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.598+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.0 size=1048576 2023-12-03T11:30:40.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.599+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.1 size=1048576 2023-12-03T11:30:40.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.600+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.2 size=76765 2023-12-03T11:30:40.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.600+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=CYWU4BY447GI tag=RRWSOZDJC27A type=write size=3222493 2023-12-03T11:30:40.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.602+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~KAMBT96R3ZBX.0 size=777378 2023-12-03T11:30:40.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.604+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=KAMBT96R3ZBX tag=QSYF5ZTJLVAK type=write size=1825954 2023-12-03T11:30:40.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.606+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.0 size=1048576 2023-12-03T11:30:40.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.608+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.1 size=1048576 2023-12-03T11:30:40.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.609+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.2 size=130524 2023-12-03T11:30:40.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.609+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=P0FRE44LJPFC tag=XSDRJXSYXXNA type=write size=3276252 2023-12-03T11:30:40.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.610+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.0 size=1048576 2023-12-03T11:30:40.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.611+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.1 size=1048576 2023-12-03T11:30:40.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.612+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~J80KR78KPJS3.2 size=654025 2023-12-03T11:30:40.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.614+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=J80KR78KPJS3 tag=YZHW3HLSM86P type=write size=3799753 2023-12-03T11:30:40.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.615+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=QLFSWPTDOSKB type=del size=3006283 2023-12-03T11:30:40.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.617+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.0 size=1048576 2023-12-03T11:30:40.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.618+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.1 size=1048576 2023-12-03T11:30:40.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.618+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~TE0571S0WHN5.2 size=593320 2023-12-03T11:30:40.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.620+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=TE0571S0WHN5 tag=E0CZ8N73N9ZW type=write size=3739048 2023-12-03T11:30:40.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.621+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=XFRDWFA1T9MP type=del size=3856082 2023-12-03T11:30:40.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.622+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=KG1YQ4I7CGZY type=del 2023-12-03T11:30:40.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.623+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=1XOVOUB5LXKT type=del size=2760007 2023-12-03T11:30:40.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.625+0000 7fc553ded780 1 < canceled operation key=WDV4CW863V33 tag=NRMLV12CBC1J type=del size=3224410 2023-12-03T11:30:40.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.628+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~4O3E8KNRPIE4.0 size=105686 2023-12-03T11:30:40.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.628+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=4O3E8KNRPIE4 tag=9R6V6BJXNF4K type=write size=1154262 2023-12-03T11:30:40.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.629+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~LRVX0Z2HORUY.0 size=334241 2023-12-03T11:30:40.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.631+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=LRVX0Z2HORUY tag=BH7OUE0U3125 type=write size=1382817 2023-12-03T11:30:40.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.632+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=ZDQY90SYQI7G type=del size=234892 2023-12-03T11:30:40.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.634+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~SJ981LPDQ9P2.0 size=1048576 2023-12-03T11:30:40.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.635+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~SJ981LPDQ9P2.1 size=650107 2023-12-03T11:30:40.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.637+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=SJ981LPDQ9P2 tag=8RBM4TSVC13I type=write size=2747259 2023-12-03T11:30:40.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.638+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=H57R4SD1F5JV type=del size=3760374 2023-12-03T11:30:40.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.639+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.0 size=1048576 2023-12-03T11:30:40.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.640+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.1 size=1048576 2023-12-03T11:30:40.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.641+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~B792IG9Y78N5.2 size=544361 2023-12-03T11:30:40.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.643+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=B792IG9Y78N5 tag=URTDB5OOBJFL type=write size=3690089 2023-12-03T11:30:40.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.647+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~DYUHEFR7XOK9.0 size=1048576 2023-12-03T11:30:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.648+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~DYUHEFR7XOK9.1 size=138282 2023-12-03T11:30:40.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.648+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=DYUHEFR7XOK9 tag=2G6F3XXSYUWI type=write size=2235434 2023-12-03T11:30:40.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.650+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=E8709FKOH6M2 type=del size=2616424 2023-12-03T11:30:40.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.651+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=2UF0B2UBL6T4 type=del size=2892109 2023-12-03T11:30:40.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.653+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=VJOWGOTIWXFJ type=write size=862675 2023-12-03T11:30:40.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.654+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=CM7BB7SWK6EQ type=del size=3278046 2023-12-03T11:30:40.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.655+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=2W43PVY0Q0DJ type=del 2023-12-03T11:30:40.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.656+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=9FNI8OF3MLCI type=del size=3796432 2023-12-03T11:30:40.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.657+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.0 size=1048576 2023-12-03T11:30:40.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.658+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.1 size=1048576 2023-12-03T11:30:40.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.659+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~JLZ020W1HTG4.2 size=432154 2023-12-03T11:30:40.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.661+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=JLZ020W1HTG4 tag=YRNDWEYXJY88 type=write size=3577882 2023-12-03T11:30:40.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.663+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=U80QLQBOWOEO type=del size=2165087 2023-12-03T11:30:40.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.664+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=IKFSIHM1GL1U type=del 2023-12-03T11:30:40.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.665+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=5SLU0RMF00WA type=del size=3177982 2023-12-03T11:30:40.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.666+0000 7fc553ded780 1 < completed write operation key=ZUNOFY5N3J3H tag=LA5J3FBTR0F5 type=write size=231149 2023-12-03T11:30:40.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.667+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=WT0W62B6OR89 type=del size=4054733 2023-12-03T11:30:40.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.669+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~0OH5EB8GKPQ6.0 size=915973 2023-12-03T11:30:40.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.670+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=0OH5EB8GKPQ6 tag=MOEDPWPQYVWP type=write size=1964549 2023-12-03T11:30:40.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.672+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=32ROW5I00RW5 type=write size=506534 2023-12-03T11:30:40.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.673+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.0 size=1048576 2023-12-03T11:30:40.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.674+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.1 size=1048576 2023-12-03T11:30:40.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.675+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~CYWU4BY447GI.2 size=76765 2023-12-03T11:30:40.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.677+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=CYWU4BY447GI tag=RRWSOZDJC27A type=write size=3222493 2023-12-03T11:30:40.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.679+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~JLK3FDOE8DA5.0 size=822296 2023-12-03T11:30:40.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.679+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=JLK3FDOE8DA5 tag=VCENWWTLCFHV type=write size=1870872 2023-12-03T11:30:40.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.680+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.0 size=1048576 2023-12-03T11:30:40.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.681+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.1 size=1048576 2023-12-03T11:30:40.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.682+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~P0FRE44LJPFC.2 size=130524 2023-12-03T11:30:40.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.684+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=P0FRE44LJPFC tag=XSDRJXSYXXNA type=write size=3276252 2023-12-03T11:30:40.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.685+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=RG8ZDWQX6KQX type=write size=918641 2023-12-03T11:30:40.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.686+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=QLFSWPTDOSKB type=del 2023-12-03T11:30:40.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.688+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=QUNBGBUJ8G1O type=del size=592910 2023-12-03T11:30:40.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.690+0000 7fc553ded780 1 < suggested remove operation key=FP2UZMDKYTD1 tag=XFRDWFA1T9MP type=del 2023-12-03T11:30:40.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.692+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=5IARWWNKTO4R type=del size=3488755 2023-12-03T11:30:40.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.693+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=1XOVOUB5LXKT type=del 2023-12-03T11:30:40.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.694+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=DYRPIQPZTMU4 type=del size=1230454 2023-12-03T11:30:40.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.695+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~4O3E8KNRPIE4.0 size=105686 2023-12-03T11:30:40.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.697+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=4O3E8KNRPIE4 tag=9R6V6BJXNF4K type=write size=1154262 2023-12-03T11:30:40.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.699+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=TU94Q4YHNN4E type=del size=2490354 2023-12-03T11:30:40.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.700+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=ZDQY90SYQI7G type=del 2023-12-03T11:30:40.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.701+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=HTL1NHVG5DNW type=del size=445960 2023-12-03T11:30:40.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.702+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=H57R4SD1F5JV type=del 2023-12-03T11:30:40.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.704+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=9R8OFAW603HQ type=del size=1602570 2023-12-03T11:30:40.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.705+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~DYUHEFR7XOK9.0 size=1048576 2023-12-03T11:30:40.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.706+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~DYUHEFR7XOK9.1 size=138282 2023-12-03T11:30:40.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.709+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=DYUHEFR7XOK9 tag=2G6F3XXSYUWI type=write size=2235434 2023-12-03T11:30:40.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.711+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.0 size=1048576 2023-12-03T11:30:40.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.712+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.1 size=1048576 2023-12-03T11:30:40.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.713+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.2 size=324194 2023-12-03T11:30:40.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.713+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=TNLFSA7INOX0 tag=03XZJA9B1VY9 type=write size=3469922 2023-12-03T11:30:40.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.716+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=2UF0B2UBL6T4 type=del size=2892109 2023-12-03T11:30:40.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.718+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.0 size=1048576 2023-12-03T11:30:40.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.719+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.1 size=1048576 2023-12-03T11:30:40.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.720+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.2 size=506004 2023-12-03T11:30:40.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.720+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=6S0BCC4AECP5 tag=TXQTS7CHQCAE type=write size=3651732 2023-12-03T11:30:40.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.722+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=CM7BB7SWK6EQ type=del 2023-12-03T11:30:40.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.724+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=YDFF3AFA42W8 type=del size=402774 2023-12-03T11:30:40.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.725+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=9FNI8OF3MLCI type=del 2023-12-03T11:30:40.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.727+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=RNNSVDR4L4RG type=del size=3953107 2023-12-03T11:30:40.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.728+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=U80QLQBOWOEO type=del 2023-12-03T11:30:40.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.729+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=CHNYUZAZIKC1 type=del size=316862 2023-12-03T11:30:40.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.732+0000 7fc553ded780 1 < suggested remove operation key=TQRZT37AB9WC tag=5SLU0RMF00WA type=del 2023-12-03T11:30:40.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.733+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=7MQX9SBXLBO7 type=del size=4050939 2023-12-03T11:30:40.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.734+0000 7fc553ded780 -1 < failed to complete operation key=L4ME9RYRR4NG tag=WT0W62B6OR89 type=del size=4054733: (22) Invalid argument 2023-12-03T11:30:40.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.736+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=9WOCYFO093LI type=del size=1028598 2023-12-03T11:30:40.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.737+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=32ROW5I00RW5 type=write size=506534 2023-12-03T11:30:40.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.738+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=Y88YBITZ1IDT type=del size=2320014 2023-12-03T11:30:40.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.739+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~JLK3FDOE8DA5.0 size=822296 2023-12-03T11:30:40.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.741+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=JLK3FDOE8DA5 tag=VCENWWTLCFHV type=write size=1870872 2023-12-03T11:30:40.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.744+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.0 size=1048576 2023-12-03T11:30:40.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.745+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.1 size=1048576 2023-12-03T11:30:40.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.747+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.2 size=843627 2023-12-03T11:30:40.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.747+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=8GW0H4ONSOKW tag=5G14MA618EHQ type=write size=3989355 2023-12-03T11:30:40.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.748+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=RG8ZDWQX6KQX type=write size=918641 2023-12-03T11:30:40.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.749+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=WXOW7WXH8CQP type=del size=2032986 2023-12-03T11:30:40.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.750+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=QUNBGBUJ8G1O type=del 2023-12-03T11:30:40.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.752+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~OQ6UCUFHXXK5.0 size=678587 2023-12-03T11:30:40.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.752+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=OQ6UCUFHXXK5 tag=VHK9QOMS4EFA type=write size=1727163 2023-12-03T11:30:40.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.754+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=5IARWWNKTO4R type=del 2023-12-03T11:30:40.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.755+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=OKGFEI2U0F2U type=del size=2328953 2023-12-03T11:30:40.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.757+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=DYRPIQPZTMU4 type=del size=1230454 2023-12-03T11:30:40.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.759+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=FK8CI0R24TTP type=write size=877001 2023-12-03T11:30:40.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.760+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=TU94Q4YHNN4E type=del 2023-12-03T11:30:40.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.761+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=U5TRWB25KORA type=del size=139712 2023-12-03T11:30:40.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.762+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=HTL1NHVG5DNW type=del 2023-12-03T11:30:40.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.765+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~SC7ZKWMNFD3G.0 size=1048576 2023-12-03T11:30:40.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.766+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~SC7ZKWMNFD3G.1 size=376011 2023-12-03T11:30:40.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.766+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=SC7ZKWMNFD3G tag=SZICEVO31OX0 type=write size=2473163 2023-12-03T11:30:40.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.767+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=9R8OFAW603HQ type=del 2023-12-03T11:30:40.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.769+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.0 size=1048576 2023-12-03T11:30:40.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.770+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.1 size=1048576 2023-12-03T11:30:40.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.771+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.2 size=75638 2023-12-03T11:30:40.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.771+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=KMKQR4BYO4OP tag=4P6R063IBK2B type=write size=3221366 2023-12-03T11:30:40.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.772+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.0 size=1048576 2023-12-03T11:30:40.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.773+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.1 size=1048576 2023-12-03T11:30:40.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.774+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~TNLFSA7INOX0.2 size=324194 2023-12-03T11:30:40.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.777+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=TNLFSA7INOX0 tag=03XZJA9B1VY9 type=write size=3469922 2023-12-03T11:30:40.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.778+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=W6281VEKZ4D3 type=del size=3282688 2023-12-03T11:30:40.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.779+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.0 size=1048576 2023-12-03T11:30:40.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.780+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.1 size=1048576 2023-12-03T11:30:40.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.781+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~6S0BCC4AECP5.2 size=506004 2023-12-03T11:30:40.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.783+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=6S0BCC4AECP5 tag=TXQTS7CHQCAE type=write size=3651732 2023-12-03T11:30:40.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.784+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=C4VR7Z7CBL6P type=write size=745142 2023-12-03T11:30:40.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.785+0000 7fc553ded780 -1 < failed to complete operation key=706CET5PSDEG tag=YDFF3AFA42W8 type=del size=402774: (22) Invalid argument 2023-12-03T11:30:40.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.787+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~VLYBOLXZEK4W.0 size=1048576 2023-12-03T11:30:40.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.788+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~VLYBOLXZEK4W.1 size=780130 2023-12-03T11:30:40.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.788+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=VLYBOLXZEK4W tag=HTST9QE8DFTS type=write size=2877282 2023-12-03T11:30:40.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.790+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=RNNSVDR4L4RG type=del size=3953107 2023-12-03T11:30:40.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.792+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~LIVTBZN9ZKJZ.0 size=1048576 2023-12-03T11:30:40.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.792+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~LIVTBZN9ZKJZ.1 size=882554 2023-12-03T11:30:40.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.792+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=LIVTBZN9ZKJZ tag=OF2GBYVD5L1T type=write size=2979706 2023-12-03T11:30:40.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.794+0000 7fc553ded780 -1 < failed to complete operation key=706CET5PSDEG tag=CHNYUZAZIKC1 type=del size=316862: (22) Invalid argument 2023-12-03T11:30:40.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.795+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=FCQUT782QUV7 type=del size=3918520 2023-12-03T11:30:40.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.799+0000 7fc553ded780 1 < suggested remove operation key=4TWM1QM1OHOF tag=7MQX9SBXLBO7 type=del 2023-12-03T11:30:40.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.801+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~RIVWW01R6IW1.0 size=1048576 2023-12-03T11:30:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.802+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~RIVWW01R6IW1.1 size=422535 2023-12-03T11:30:40.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.802+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=RIVWW01R6IW1 tag=DLSE7Y77HBBJ type=write size=2519687 2023-12-03T11:30:40.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.803+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=9WOCYFO093LI type=del 2023-12-03T11:30:40.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.804+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=JOZRSYT125QY type=write size=287892 2023-12-03T11:30:40.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.806+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=Y88YBITZ1IDT type=del 2023-12-03T11:30:40.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.807+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=IX0D11I3EWJX type=del size=3093065 2023-12-03T11:30:40.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.807+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.0 size=1048576 2023-12-03T11:30:40.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.808+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.1 size=1048576 2023-12-03T11:30:40.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.809+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~8GW0H4ONSOKW.2 size=843627 2023-12-03T11:30:40.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.811+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=8GW0H4ONSOKW tag=5G14MA618EHQ type=write size=3989355 2023-12-03T11:30:40.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.812+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=AT29KZNRSR86 type=write size=329776 2023-12-03T11:30:40.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.813+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=WXOW7WXH8CQP type=del 2023-12-03T11:30:40.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.814+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=0COOTRAP3TKI type=del size=2606970 2023-12-03T11:30:40.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.816+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~OQ6UCUFHXXK5.0 size=678587 2023-12-03T11:30:40.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.818+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=OQ6UCUFHXXK5 tag=VHK9QOMS4EFA type=write size=1727163 2023-12-03T11:30:40.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.819+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=TUPAJUNF2N8B type=del size=1311088 2023-12-03T11:30:40.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.821+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=OKGFEI2U0F2U type=del 2023-12-03T11:30:40.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.822+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=UMMA9FLXX3EF type=del size=1314412 2023-12-03T11:30:40.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.824+0000 7fc553ded780 1 < completed write operation key=WLTMDGRBF4C9 tag=FK8CI0R24TTP type=write size=877001 2023-12-03T11:30:40.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.825+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=OWRT21ZPK1TZ type=del size=3851114 2023-12-03T11:30:40.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.826+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=U5TRWB25KORA type=del 2023-12-03T11:30:40.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.828+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=8HESMOD5033R type=del size=2787699 2023-12-03T11:30:40.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.829+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~SC7ZKWMNFD3G.0 size=1048576 2023-12-03T11:30:40.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.830+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~SC7ZKWMNFD3G.1 size=376011 2023-12-03T11:30:40.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.833+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=SC7ZKWMNFD3G tag=SZICEVO31OX0 type=write size=2473163 2023-12-03T11:30:40.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.834+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=OOQIIZD6ODSH type=del size=2654148 2023-12-03T11:30:40.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.835+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.0 size=1048576 2023-12-03T11:30:40.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.836+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.1 size=1048576 2023-12-03T11:30:40.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.837+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~KMKQR4BYO4OP.2 size=75638 2023-12-03T11:30:40.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.840+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=KMKQR4BYO4OP tag=4P6R063IBK2B type=write size=3221366 2023-12-03T11:30:40.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.841+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=F7JKPNSASB7Z type=del size=3002284 2023-12-03T11:30:40.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.842+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=W6281VEKZ4D3 type=del 2023-12-03T11:30:40.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.843+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=RLJUMHIKGYCS type=del size=3801537 2023-12-03T11:30:40.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.845+0000 7fc553ded780 1 < completed write operation key=C2TPF9Q5Z7XQ tag=C4VR7Z7CBL6P type=write size=745142 2023-12-03T11:30:40.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.847+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~BVB7DY0Y80FI.0 size=1048576 2023-12-03T11:30:40.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.848+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~BVB7DY0Y80FI.1 size=508205 2023-12-03T11:30:40.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.848+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=BVB7DY0Y80FI tag=YXWO69E2Y5IJ type=write size=2605357 2023-12-03T11:30:40.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.849+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~VLYBOLXZEK4W.0 size=1048576 2023-12-03T11:30:40.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.850+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~VLYBOLXZEK4W.1 size=780130 2023-12-03T11:30:40.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.852+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=VLYBOLXZEK4W tag=HTST9QE8DFTS type=write size=2877282 2023-12-03T11:30:40.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.853+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=TDRHBRJASH8T type=del size=1821066 2023-12-03T11:30:40.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.855+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~LIVTBZN9ZKJZ.0 size=1048576 2023-12-03T11:30:40.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.856+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~LIVTBZN9ZKJZ.1 size=882554 2023-12-03T11:30:40.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.858+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=LIVTBZN9ZKJZ tag=OF2GBYVD5L1T type=write size=2979706 2023-12-03T11:30:40.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.859+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=6SJS1D1A43BA type=del size=1349911 2023-12-03T11:30:40.861 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.860+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=FCQUT782QUV7 type=del 2023-12-03T11:30:40.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.862+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=AT1EF4U5KSKB type=del size=1644190 2023-12-03T11:30:40.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.864+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~RIVWW01R6IW1.0 size=1048576 2023-12-03T11:30:40.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.865+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~RIVWW01R6IW1.1 size=422535 2023-12-03T11:30:40.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.867+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=RIVWW01R6IW1 tag=DLSE7Y77HBBJ type=write size=2519687 2023-12-03T11:30:40.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.868+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=9BT88GJAWMRD type=del size=2198190 2023-12-03T11:30:40.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.870+0000 7fc553ded780 1 < suggested update operation key=WOKG1DISIKK0 tag=JOZRSYT125QY type=write size=287892 2023-12-03T11:30:40.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.872+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=V31CVHSBG2XX type=del size=1308359 2023-12-03T11:30:40.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.873+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=IX0D11I3EWJX type=del 2023-12-03T11:30:40.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.874+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=O52WUDREK6OJ type=del size=1255881 2023-12-03T11:30:40.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.875+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=AT29KZNRSR86 type=write size=329776 2023-12-03T11:30:40.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.878+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~VLRYD5VW5QBC.0 size=1048576 2023-12-03T11:30:40.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.879+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~VLRYD5VW5QBC.1 size=962806 2023-12-03T11:30:40.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.879+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=VLRYD5VW5QBC tag=ODRI4MHSIHKP type=write size=3059958 2023-12-03T11:30:40.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.880+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=0COOTRAP3TKI type=del 2023-12-03T11:30:40.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.882+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~W9JO4V4PENO6.0 size=453419 2023-12-03T11:30:40.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.882+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=W9JO4V4PENO6 tag=XD1VR2BRV5KH type=write size=1501995 2023-12-03T11:30:40.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.883+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=TUPAJUNF2N8B type=del 2023-12-03T11:30:40.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.887+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.0 size=1048576 2023-12-03T11:30:40.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.888+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.1 size=1048576 2023-12-03T11:30:40.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.889+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.2 size=86649 2023-12-03T11:30:40.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.889+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=QLKKI5GTQPDB tag=1YK8FJQEROGH type=write size=3232377 2023-12-03T11:30:40.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.892+0000 7fc553ded780 1 < suggested update operation key=RO0KGI2QMA71 tag=UMMA9FLXX3EF type=del size=1314412 2023-12-03T11:30:40.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.893+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=JVYUD34OCFZM type=del size=3826793 2023-12-03T11:30:40.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.895+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=OWRT21ZPK1TZ type=del 2023-12-03T11:30:40.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.897+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~DTVI01I3SQFK.0 size=613766 2023-12-03T11:30:40.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.897+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=DTVI01I3SQFK tag=RSJQMXUWUWGJ type=write size=1662342 2023-12-03T11:30:40.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.899+0000 7fc553ded780 1 < suggested update operation key=ZUNOFY5N3J3H tag=8HESMOD5033R type=del size=2787699 2023-12-03T11:30:40.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.901+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~HSEERZRBVD7P.0 size=1048576 2023-12-03T11:30:40.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.902+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~HSEERZRBVD7P.1 size=296641 2023-12-03T11:30:40.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.902+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=HSEERZRBVD7P tag=GEF7XTFPBXYM type=write size=2393793 2023-12-03T11:30:40.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.904+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=OOQIIZD6ODSH type=del 2023-12-03T11:30:40.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.905+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=IU3P3CHUBPNM type=write size=128764 2023-12-03T11:30:40.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.906+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=F7JKPNSASB7Z type=del 2023-12-03T11:30:40.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.908+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=Q3PEEEJGK189 type=del size=1156773 2023-12-03T11:30:40.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.910+0000 7fc553ded780 1 < suggested remove operation key=WDV4CW863V33 tag=RLJUMHIKGYCS type=del 2023-12-03T11:30:40.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.911+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=569QPXDR0408 type=del size=1718739 2023-12-03T11:30:40.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.912+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~BVB7DY0Y80FI.0 size=1048576 2023-12-03T11:30:40.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.912+0000 7fc553ded780 1 < canceled part key=_multipart_588S2ASN9I0W.2~BVB7DY0Y80FI.1 size=508205 2023-12-03T11:30:40.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.914+0000 7fc553ded780 1 < canceled multipart upload key=588S2ASN9I0W upload=BVB7DY0Y80FI tag=YXWO69E2Y5IJ type=write size=2605357 2023-12-03T11:30:40.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.916+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=Y9OJ2STW6KIN type=write size=816625 2023-12-03T11:30:40.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.917+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=TDRHBRJASH8T type=del 2023-12-03T11:30:40.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.918+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=0EE0BCSLXK2L type=del size=1916033 2023-12-03T11:30:40.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.919+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=6SJS1D1A43BA type=del size=1349911: (22) Invalid argument 2023-12-03T11:30:40.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.921+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=2ZD2X726LC61 type=del size=1886891 2023-12-03T11:30:40.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.923+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=AT1EF4U5KSKB type=del size=1644190 2023-12-03T11:30:40.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.925+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~LBRB7F4SUQFY.0 size=507183 2023-12-03T11:30:40.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.925+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=LBRB7F4SUQFY tag=THJR77G2RVPC type=write size=1555759 2023-12-03T11:30:40.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.927+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=9BT88GJAWMRD type=del 2023-12-03T11:30:40.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.929+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.0 size=1048576 2023-12-03T11:30:40.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.930+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.1 size=1048576 2023-12-03T11:30:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.931+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.2 size=594678 2023-12-03T11:30:40.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.931+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=5OT05FW7VF8Y tag=KVP3E8RR6NOS type=write size=3740406 2023-12-03T11:30:40.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.932+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=V31CVHSBG2XX type=del 2023-12-03T11:30:40.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.933+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=314K3BNLY0I9 type=del size=2570482 2023-12-03T11:30:40.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.935+0000 7fc553ded780 -1 < failed to complete operation key=RO0KGI2QMA71 tag=O52WUDREK6OJ type=del size=1255881: (22) Invalid argument 2023-12-03T11:30:40.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.937+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~B7KKX43MWK4F.0 size=405890 2023-12-03T11:30:40.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.937+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=B7KKX43MWK4F tag=KN3W3HK03QRA type=write size=1454466 2023-12-03T11:30:40.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.938+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~VLRYD5VW5QBC.0 size=1048576 2023-12-03T11:30:40.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.939+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~VLRYD5VW5QBC.1 size=962806 2023-12-03T11:30:40.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.941+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=VLRYD5VW5QBC tag=ODRI4MHSIHKP type=write size=3059958 2023-12-03T11:30:40.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.943+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~CSNJ0A7SK799.0 size=1048576 2023-12-03T11:30:40.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.944+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~CSNJ0A7SK799.1 size=562975 2023-12-03T11:30:40.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.944+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=CSNJ0A7SK799 tag=4J11RG48FDAM type=write size=2660127 2023-12-03T11:30:40.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.946+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~W9JO4V4PENO6.0 size=453419 2023-12-03T11:30:40.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.948+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=W9JO4V4PENO6 tag=XD1VR2BRV5KH type=write size=1501995 2023-12-03T11:30:40.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.950+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~Y5GKLF02FI26.0 size=235151 2023-12-03T11:30:40.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.950+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=Y5GKLF02FI26 tag=ZF6Q3AG1IJP9 type=write size=1283727 2023-12-03T11:30:40.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.951+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.0 size=1048576 2023-12-03T11:30:40.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.952+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.1 size=1048576 2023-12-03T11:30:40.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.953+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~QLKKI5GTQPDB.2 size=86649 2023-12-03T11:30:40.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.955+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=QLKKI5GTQPDB tag=1YK8FJQEROGH type=write size=3232377 2023-12-03T11:30:40.958 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.957+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=13CG79PUAG2C type=del size=3239630 2023-12-03T11:30:40.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.960+0000 7fc553ded780 1 < suggested update operation key=C2TPF9Q5Z7XQ tag=JVYUD34OCFZM type=del size=3826793 2023-12-03T11:30:40.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.961+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=81HBEAX5MFHD type=del size=1104805 2023-12-03T11:30:40.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.963+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~DTVI01I3SQFK.0 size=613766 2023-12-03T11:30:40.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.965+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=DTVI01I3SQFK tag=RSJQMXUWUWGJ type=write size=1662342 2023-12-03T11:30:40.967 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.967+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=QRD69IXP9978 type=del size=1847126 2023-12-03T11:30:40.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.968+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~HSEERZRBVD7P.0 size=1048576 2023-12-03T11:30:40.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.969+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~HSEERZRBVD7P.1 size=296641 2023-12-03T11:30:40.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.971+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=HSEERZRBVD7P tag=GEF7XTFPBXYM type=write size=2393793 2023-12-03T11:30:40.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.972+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=1LQV1YU0DHI1 type=del size=3551154 2023-12-03T11:30:40.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.973+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=IU3P3CHUBPNM type=write size=128764 2023-12-03T11:30:40.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.975+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=EOT5QIGK2RYK type=del size=3479706 2023-12-03T11:30:40.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.977+0000 7fc553ded780 1 < suggested remove operation key=J664UUE697Q8 tag=Q3PEEEJGK189 type=del 2023-12-03T11:30:40.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.980+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.0 size=1048576 2023-12-03T11:30:40.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.981+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.1 size=1048576 2023-12-03T11:30:40.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.982+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.2 size=12500 2023-12-03T11:30:40.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.983+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=89V8KAVT8AXL tag=LTAUSR3INH01 type=write size=3158228 2023-12-03T11:30:40.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.984+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=569QPXDR0408 type=del 2023-12-03T11:30:40.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.985+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=IDHIIG2G96NL type=del size=1192246 2023-12-03T11:30:40.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.987+0000 7fc553ded780 1 < suggested update operation key=C2TPF9Q5Z7XQ tag=Y9OJ2STW6KIN type=write size=816625 2023-12-03T11:30:40.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.990+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.0 size=1048576 2023-12-03T11:30:40.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.991+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.1 size=1048576 2023-12-03T11:30:40.993 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.992+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.2 size=529909 2023-12-03T11:30:40.993 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.992+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=OC5JBI8LOHUF tag=NJMQ0ZDOQET7 type=write size=3675637 2023-12-03T11:30:40.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.993+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=0EE0BCSLXK2L type=del 2023-12-03T11:30:40.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.995+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~X7IGJAG4BRZH.0 size=946904 2023-12-03T11:30:40.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.995+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=X7IGJAG4BRZH tag=F1KSU5EL0OMQ type=write size=1995480 2023-12-03T11:30:40.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.996+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=2ZD2X726LC61 type=del 2023-12-03T11:30:40.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.998+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~ZG68B7JNCOMZ.0 size=1001170 2023-12-03T11:30:40.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:40.998+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=ZG68B7JNCOMZ tag=X2XA4VY47L35 type=write size=2049746 2023-12-03T11:30:41.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.000+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~LBRB7F4SUQFY.0 size=507183 2023-12-03T11:30:41.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.002+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=LBRB7F4SUQFY tag=THJR77G2RVPC type=write size=1555759 2023-12-03T11:30:41.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.004+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~H1N2VPD5S79Q.0 size=649582 2023-12-03T11:30:41.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.004+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=H1N2VPD5S79Q tag=QF9GH56CPPNY type=write size=1698158 2023-12-03T11:30:41.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.005+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.0 size=1048576 2023-12-03T11:30:41.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.006+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.1 size=1048576 2023-12-03T11:30:41.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.007+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~5OT05FW7VF8Y.2 size=594678 2023-12-03T11:30:41.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.009+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=5OT05FW7VF8Y tag=KVP3E8RR6NOS type=write size=3740406 2023-12-03T11:30:41.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.013+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.0 size=1048576 2023-12-03T11:30:41.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.014+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.1 size=1048576 2023-12-03T11:30:41.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.015+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.2 size=410189 2023-12-03T11:30:41.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.015+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=TLQA02VM1CNG tag=HJGC55QYS4QD type=write size=3555917 2023-12-03T11:30:41.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.016+0000 7fc553ded780 1 < canceled operation key=PL22GPUFN3O0 tag=314K3BNLY0I9 type=del size=2570482 2023-12-03T11:30:41.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.018+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~F4NIBXP3K084.0 size=1048576 2023-12-03T11:30:41.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.019+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~F4NIBXP3K084.1 size=903267 2023-12-03T11:30:41.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.019+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=F4NIBXP3K084 tag=XE25RUFWOREY type=write size=3000419 2023-12-03T11:30:41.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.021+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~B7KKX43MWK4F.0 size=405890 2023-12-03T11:30:41.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.023+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=B7KKX43MWK4F tag=KN3W3HK03QRA type=write size=1454466 2023-12-03T11:30:41.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.025+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~BCXASQK3DAV3.0 size=1048576 2023-12-03T11:30:41.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.026+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~BCXASQK3DAV3.1 size=327541 2023-12-03T11:30:41.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.026+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=BCXASQK3DAV3 tag=PP8VOD0ZLRX4 type=write size=2424693 2023-12-03T11:30:41.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.026+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~CSNJ0A7SK799.0 size=1048576 2023-12-03T11:30:41.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.027+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~CSNJ0A7SK799.1 size=562975 2023-12-03T11:30:41.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.029+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=CSNJ0A7SK799 tag=4J11RG48FDAM type=write size=2660127 2023-12-03T11:30:41.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.031+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.0 size=1048576 2023-12-03T11:30:41.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.032+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.1 size=1048576 2023-12-03T11:30:41.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.033+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.2 size=1002271 2023-12-03T11:30:41.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.033+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=GEFDIU33G4JF tag=394WGL8A2M8B type=write size=4147999 2023-12-03T11:30:41.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.034+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~Y5GKLF02FI26.0 size=235151 2023-12-03T11:30:41.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.036+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=Y5GKLF02FI26 tag=ZF6Q3AG1IJP9 type=write size=1283727 2023-12-03T11:30:41.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.039+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~84BNE1VWW3DO.0 size=937913 2023-12-03T11:30:41.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.039+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=84BNE1VWW3DO tag=XP2P60FSA1K3 type=write size=1986489 2023-12-03T11:30:41.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.040+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=13CG79PUAG2C type=del 2023-12-03T11:30:41.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.041+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=S2E8Y5MB9B3N type=write size=814374 2023-12-03T11:30:41.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.042+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=81HBEAX5MFHD type=del 2023-12-03T11:30:41.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.044+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=WYXV9G9IJ6QY type=del size=1004902 2023-12-03T11:30:41.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.045+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=QRD69IXP9978 type=del 2023-12-03T11:30:41.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.046+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=252IWTJ00X8A type=del size=1038790 2023-12-03T11:30:41.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.047+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=1LQV1YU0DHI1 type=del 2023-12-03T11:30:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.050+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~O4GYRHABJJPM.0 size=1048485 2023-12-03T11:30:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.050+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=O4GYRHABJJPM tag=2MU01Q5HY127 type=write size=2097061 2023-12-03T11:30:41.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.051+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=EOT5QIGK2RYK type=del 2023-12-03T11:30:41.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.052+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=3A9EAOW1KPG1 type=write size=667106 2023-12-03T11:30:41.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.054+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.0 size=1048576 2023-12-03T11:30:41.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.055+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.1 size=1048576 2023-12-03T11:30:41.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.056+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~89V8KAVT8AXL.2 size=12500 2023-12-03T11:30:41.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.058+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=89V8KAVT8AXL tag=LTAUSR3INH01 type=write size=3158228 2023-12-03T11:30:41.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.059+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=WAPLICOWEEJI type=write size=645034 2023-12-03T11:30:41.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.060+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=IDHIIG2G96NL type=del 2023-12-03T11:30:41.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.061+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=EUWQPN27I0AP type=del size=2771567 2023-12-03T11:30:41.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.063+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.0 size=1048576 2023-12-03T11:30:41.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.064+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.1 size=1048576 2023-12-03T11:30:41.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.065+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~OC5JBI8LOHUF.2 size=529909 2023-12-03T11:30:41.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.067+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=OC5JBI8LOHUF tag=NJMQ0ZDOQET7 type=write size=3675637 2023-12-03T11:30:41.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.068+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=UHFM0BRYB7DF type=del size=2609453 2023-12-03T11:30:41.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.069+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~X7IGJAG4BRZH.0 size=946904 2023-12-03T11:30:41.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.071+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=X7IGJAG4BRZH tag=F1KSU5EL0OMQ type=write size=1995480 2023-12-03T11:30:41.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.073+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~09JK575ZXKYK.0 size=1048576 2023-12-03T11:30:41.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.074+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~09JK575ZXKYK.1 size=419708 2023-12-03T11:30:41.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.074+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=09JK575ZXKYK tag=HX0IE1YQ01UH type=write size=2516860 2023-12-03T11:30:41.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.075+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~ZG68B7JNCOMZ.0 size=1001170 2023-12-03T11:30:41.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.077+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=ZG68B7JNCOMZ tag=X2XA4VY47L35 type=write size=2049746 2023-12-03T11:30:41.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.078+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=13YN4C2W149O type=del size=2422847 2023-12-03T11:30:41.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.080+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~H1N2VPD5S79Q.0 size=649582 2023-12-03T11:30:41.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.082+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=H1N2VPD5S79Q tag=QF9GH56CPPNY type=write size=1698158 2023-12-03T11:30:41.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.084+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~34U7Y7DB7D8Z.0 size=325167 2023-12-03T11:30:41.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.084+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=34U7Y7DB7D8Z tag=G0MU133R1AAZ type=write size=1373743 2023-12-03T11:30:41.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.085+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.0 size=1048576 2023-12-03T11:30:41.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.086+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.1 size=1048576 2023-12-03T11:30:41.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.087+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~TLQA02VM1CNG.2 size=410189 2023-12-03T11:30:41.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.089+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=TLQA02VM1CNG tag=HJGC55QYS4QD type=write size=3555917 2023-12-03T11:30:41.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.091+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=GPXWQ7DZ1HOS type=del size=4047049 2023-12-03T11:30:41.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.092+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~F4NIBXP3K084.0 size=1048576 2023-12-03T11:30:41.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.093+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~F4NIBXP3K084.1 size=903267 2023-12-03T11:30:41.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.096+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=F4NIBXP3K084 tag=XE25RUFWOREY type=write size=3000419 2023-12-03T11:30:41.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.097+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=JQYKL8CIAHVV type=del size=3929322 2023-12-03T11:30:41.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.098+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~BCXASQK3DAV3.0 size=1048576 2023-12-03T11:30:41.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.099+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~BCXASQK3DAV3.1 size=327541 2023-12-03T11:30:41.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.101+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=BCXASQK3DAV3 tag=PP8VOD0ZLRX4 type=write size=2424693 2023-12-03T11:30:41.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.102+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=9613C5SOMIPD type=del size=1532452 2023-12-03T11:30:41.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.103+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.0 size=1048576 2023-12-03T11:30:41.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.103+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.1 size=1048576 2023-12-03T11:30:41.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.104+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~GEFDIU33G4JF.2 size=1002271 2023-12-03T11:30:41.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.106+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=GEFDIU33G4JF tag=394WGL8A2M8B type=write size=4147999 2023-12-03T11:30:41.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.108+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~J55RG0884M8I.0 size=1048576 2023-12-03T11:30:41.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.109+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~J55RG0884M8I.1 size=1016017 2023-12-03T11:30:41.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.109+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=J55RG0884M8I tag=4726L1T1B5PN type=write size=3113169 2023-12-03T11:30:41.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.111+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~84BNE1VWW3DO.0 size=937913 2023-12-03T11:30:41.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.113+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=84BNE1VWW3DO tag=XP2P60FSA1K3 type=write size=1986489 2023-12-03T11:30:41.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.114+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=QN93JI9CSEK8 type=del size=3017894 2023-12-03T11:30:41.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.115+0000 7fc553ded780 1 < completed write operation key=ICDF1EYNI5IU tag=S2E8Y5MB9B3N type=write size=814374 2023-12-03T11:30:41.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.118+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.0 size=1048576 2023-12-03T11:30:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.119+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.1 size=1048576 2023-12-03T11:30:41.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.120+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.2 size=888341 2023-12-03T11:30:41.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.120+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=BQ80POH08Q9Q tag=Q159EISUHSJ7 type=write size=4034069 2023-12-03T11:30:41.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.121+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=WYXV9G9IJ6QY type=del 2023-12-03T11:30:41.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.124+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.0 size=1048576 2023-12-03T11:30:41.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.125+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.1 size=1048576 2023-12-03T11:30:41.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.126+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.2 size=396442 2023-12-03T11:30:41.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.126+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=3YTSL4V5HD01 tag=N32K14VAPG9F type=write size=3542170 2023-12-03T11:30:41.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.128+0000 7fc553ded780 1 < suggested update operation key=JUYA4LXEQ4US tag=252IWTJ00X8A type=del size=1038790 2023-12-03T11:30:41.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.129+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=3HEWMU427UZI type=del size=1316686 2023-12-03T11:30:41.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.130+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~O4GYRHABJJPM.0 size=1048485 2023-12-03T11:30:41.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.133+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=O4GYRHABJJPM tag=2MU01Q5HY127 type=write size=2097061 2023-12-03T11:30:41.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.135+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.0 size=1048576 2023-12-03T11:30:41.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.136+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.1 size=1048576 2023-12-03T11:30:41.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.137+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.2 size=45798 2023-12-03T11:30:41.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.137+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=8WLTCX04QO2P tag=XDGP9HQNU83I type=write size=3191526 2023-12-03T11:30:41.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.138+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=3A9EAOW1KPG1 type=write size=667106 2023-12-03T11:30:41.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.140+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=72FU1DHWL79H type=write size=922118 2023-12-03T11:30:41.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.141+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=WAPLICOWEEJI type=write size=645034 2023-12-03T11:30:41.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.143+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.0 size=1048576 2023-12-03T11:30:41.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.144+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.1 size=1048576 2023-12-03T11:30:41.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.145+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.2 size=410214 2023-12-03T11:30:41.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.145+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=JJQUF08U3DP7 tag=9GHFI8A22SU1 type=write size=3555942 2023-12-03T11:30:41.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.148+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=EUWQPN27I0AP type=del size=2771567 2023-12-03T11:30:41.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.149+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=GQDTX2RWMAGR type=del size=2719585 2023-12-03T11:30:41.151 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.150+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=UHFM0BRYB7DF type=del 2023-12-03T11:30:41.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.153+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~83VWX8KIEYJQ.0 size=529074 2023-12-03T11:30:41.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.153+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=83VWX8KIEYJQ tag=H4XGCQ33O03H type=write size=1577650 2023-12-03T11:30:41.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.154+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~09JK575ZXKYK.0 size=1048576 2023-12-03T11:30:41.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.155+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~09JK575ZXKYK.1 size=419708 2023-12-03T11:30:41.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.157+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=09JK575ZXKYK tag=HX0IE1YQ01UH type=write size=2516860 2023-12-03T11:30:41.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.158+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=BA8P6FJOXO0J type=del size=827887 2023-12-03T11:30:41.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.159+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=13YN4C2W149O type=del 2023-12-03T11:30:41.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.161+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=WZ50Y13DW3AE type=del size=1717100 2023-12-03T11:30:41.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.162+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~34U7Y7DB7D8Z.0 size=325167 2023-12-03T11:30:41.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.164+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=34U7Y7DB7D8Z tag=G0MU133R1AAZ type=write size=1373743 2023-12-03T11:30:41.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.167+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.0 size=1048576 2023-12-03T11:30:41.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.168+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.1 size=1048576 2023-12-03T11:30:41.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.169+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.2 size=1006046 2023-12-03T11:30:41.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.169+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=DWYX19NZ6YTB tag=1LYZP8FJNHM9 type=write size=4151774 2023-12-03T11:30:41.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.170+0000 7fc553ded780 -1 < failed to complete operation key=0H5OZMBMLQFJ tag=GPXWQ7DZ1HOS type=del size=4047049: (22) Invalid argument 2023-12-03T11:30:41.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.171+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=1ABVNQFFW60P type=del size=751741 2023-12-03T11:30:41.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.172+0000 7fc553ded780 -1 < failed to complete operation key=JUYA4LXEQ4US tag=JQYKL8CIAHVV type=del size=3929322: (22) Invalid argument 2023-12-03T11:30:41.176 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.175+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~D910XI6IJGA1.0 size=1048576 2023-12-03T11:30:41.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.176+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~D910XI6IJGA1.1 size=452828 2023-12-03T11:30:41.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.176+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=D910XI6IJGA1 tag=AFYLK2434833 type=write size=2549980 2023-12-03T11:30:41.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.177+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=9613C5SOMIPD type=del size=1532452 2023-12-03T11:30:41.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.179+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=19L36PX4W9AM type=del size=2298723 2023-12-03T11:30:41.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.180+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~J55RG0884M8I.0 size=1048576 2023-12-03T11:30:41.182 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.181+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~J55RG0884M8I.1 size=1016017 2023-12-03T11:30:41.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.184+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=J55RG0884M8I tag=4726L1T1B5PN type=write size=3113169 2023-12-03T11:30:41.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.185+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=USWNMHF1UB5Y type=del size=2074046 2023-12-03T11:30:41.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.187+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=QN93JI9CSEK8 type=del 2023-12-03T11:30:41.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.189+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~K5Y5HZCECO7T.0 size=547087 2023-12-03T11:30:41.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.189+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=K5Y5HZCECO7T tag=GRXM01JMZBWX type=write size=1595663 2023-12-03T11:30:41.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.190+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.0 size=1048576 2023-12-03T11:30:41.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.191+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.1 size=1048576 2023-12-03T11:30:41.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.192+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~BQ80POH08Q9Q.2 size=888341 2023-12-03T11:30:41.195 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.194+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=BQ80POH08Q9Q tag=Q159EISUHSJ7 type=write size=4034069 2023-12-03T11:30:41.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.196+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=974A79PYZ3S9 type=write size=136555 2023-12-03T11:30:41.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.197+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.0 size=1048576 2023-12-03T11:30:41.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.198+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.1 size=1048576 2023-12-03T11:30:41.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.199+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~3YTSL4V5HD01.2 size=396442 2023-12-03T11:30:41.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.202+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=3YTSL4V5HD01 tag=N32K14VAPG9F type=write size=3542170 2023-12-03T11:30:41.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.204+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.0 size=1048576 2023-12-03T11:30:41.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.205+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.1 size=1048576 2023-12-03T11:30:41.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.206+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.2 size=103727 2023-12-03T11:30:41.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.206+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=EPGWUXV3C3LR tag=0DR8W4QNX7B3 type=write size=3249455 2023-12-03T11:30:41.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.207+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=3HEWMU427UZI type=del 2023-12-03T11:30:41.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.208+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=A12TQ29DNSRX type=del size=2821210 2023-12-03T11:30:41.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.210+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.0 size=1048576 2023-12-03T11:30:41.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.212+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.1 size=1048576 2023-12-03T11:30:41.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.214+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~8WLTCX04QO2P.2 size=45798 2023-12-03T11:30:41.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.217+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=8WLTCX04QO2P tag=XDGP9HQNU83I type=write size=3191526 2023-12-03T11:30:41.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.220+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.0 size=1048576 2023-12-03T11:30:41.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.220+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.1 size=1048576 2023-12-03T11:30:41.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.221+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.2 size=985696 2023-12-03T11:30:41.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.221+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=UZCZE58VU1Z6 tag=Y6BY3YQ69LG2 type=write size=4131424 2023-12-03T11:30:41.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.223+0000 7fc553ded780 1 < completed write operation key=ICDF1EYNI5IU tag=72FU1DHWL79H type=write size=922118 2023-12-03T11:30:41.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.224+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=F10WTZFSUVWL type=del size=3378598 2023-12-03T11:30:41.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.225+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.0 size=1048576 2023-12-03T11:30:41.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.226+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.1 size=1048576 2023-12-03T11:30:41.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.227+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~JJQUF08U3DP7.2 size=410214 2023-12-03T11:30:41.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.229+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=JJQUF08U3DP7 tag=9GHFI8A22SU1 type=write size=3555942 2023-12-03T11:30:41.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.230+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=LZ9R1VV1S2J2 type=del size=2713725 2023-12-03T11:30:41.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.232+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=GQDTX2RWMAGR type=del 2023-12-03T11:30:41.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.233+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=QZRBB476497T type=del size=412326 2023-12-03T11:30:41.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.233+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~83VWX8KIEYJQ.0 size=529074 2023-12-03T11:30:41.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.235+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=83VWX8KIEYJQ tag=H4XGCQ33O03H type=write size=1577650 2023-12-03T11:30:41.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.237+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~1BRROK27YMHC.0 size=1048576 2023-12-03T11:30:41.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.238+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~1BRROK27YMHC.1 size=404670 2023-12-03T11:30:41.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.238+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=1BRROK27YMHC tag=4IUF3BP9SRGM type=write size=2501822 2023-12-03T11:30:41.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.239+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=BA8P6FJOXO0J type=del 2023-12-03T11:30:41.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.240+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=S4G455NKTBPQ type=write size=880511 2023-12-03T11:30:41.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.242+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=WZ50Y13DW3AE type=del 2023-12-03T11:30:41.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.244+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.0 size=1048576 2023-12-03T11:30:41.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.245+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.1 size=1048576 2023-12-03T11:30:41.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.246+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.2 size=646029 2023-12-03T11:30:41.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.246+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=FNXF9RWG7VNF tag=6O5M302JKE07 type=write size=3791757 2023-12-03T11:30:41.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.247+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.0 size=1048576 2023-12-03T11:30:41.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.248+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.1 size=1048576 2023-12-03T11:30:41.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.249+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~DWYX19NZ6YTB.2 size=1006046 2023-12-03T11:30:41.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.252+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=DWYX19NZ6YTB tag=1LYZP8FJNHM9 type=write size=4151774 2023-12-03T11:30:41.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.253+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=DGDEFO36YSFU type=del size=3631010 2023-12-03T11:30:41.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.254+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=1ABVNQFFW60P type=del 2023-12-03T11:30:41.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.256+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=66PW16082187 type=del size=2365127 2023-12-03T11:30:41.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.257+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~D910XI6IJGA1.0 size=1048576 2023-12-03T11:30:41.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.258+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~D910XI6IJGA1.1 size=452828 2023-12-03T11:30:41.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.260+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=D910XI6IJGA1 tag=AFYLK2434833 type=write size=2549980 2023-12-03T11:30:41.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.263+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~1JU7SI5A9SUL.0 size=956916 2023-12-03T11:30:41.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.263+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=1JU7SI5A9SUL tag=9RRWS7FBPZL4 type=write size=2005492 2023-12-03T11:30:41.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.264+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=19L36PX4W9AM type=del 2023-12-03T11:30:41.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.265+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=0RJM7F2S7VCN type=del size=1829268 2023-12-03T11:30:41.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.266+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=USWNMHF1UB5Y type=del 2023-12-03T11:30:41.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.269+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~WXWY20WKY702.0 size=946797 2023-12-03T11:30:41.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.269+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=WXWY20WKY702 tag=UM4PZTF0EXRZ type=write size=1995373 2023-12-03T11:30:41.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.269+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~K5Y5HZCECO7T.0 size=547087 2023-12-03T11:30:41.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.271+0000 7fc553ded780 1 < canceled multipart upload key=EK7V6U9DSLD0 upload=K5Y5HZCECO7T tag=GRXM01JMZBWX type=write size=1595663 2023-12-03T11:30:41.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.272+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=M0JI2BFD6TX1 type=write size=289508 2023-12-03T11:30:41.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.273+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=974A79PYZ3S9 type=write size=136555 2023-12-03T11:30:41.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.276+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~20C8LH5FOFSL.0 size=1048576 2023-12-03T11:30:41.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.277+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~20C8LH5FOFSL.1 size=871878 2023-12-03T11:30:41.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.277+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=20C8LH5FOFSL tag=IMW6870G2Z1G type=write size=2969030 2023-12-03T11:30:41.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.278+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.0 size=1048576 2023-12-03T11:30:41.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.279+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.1 size=1048576 2023-12-03T11:30:41.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.280+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~EPGWUXV3C3LR.2 size=103727 2023-12-03T11:30:41.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.283+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=EPGWUXV3C3LR tag=0DR8W4QNX7B3 type=write size=3249455 2023-12-03T11:30:41.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.285+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.0 size=1048576 2023-12-03T11:30:41.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.286+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.1 size=1048576 2023-12-03T11:30:41.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.287+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.2 size=776121 2023-12-03T11:30:41.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.287+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=J7N2XXEBZ94V tag=DN83FUSJNO9F type=write size=3921849 2023-12-03T11:30:41.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.289+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=A12TQ29DNSRX type=del 2023-12-03T11:30:41.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.291+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JCEJA3LV8ZL2.0 size=787450 2023-12-03T11:30:41.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.291+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=JCEJA3LV8ZL2 tag=Z6UKOUJR2DVK type=write size=1836026 2023-12-03T11:30:41.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.292+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.0 size=1048576 2023-12-03T11:30:41.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.294+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.1 size=1048576 2023-12-03T11:30:41.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.295+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UZCZE58VU1Z6.2 size=985696 2023-12-03T11:30:41.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.297+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=UZCZE58VU1Z6 tag=Y6BY3YQ69LG2 type=write size=4131424 2023-12-03T11:30:41.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.298+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=DW8QBT2PUKS2 type=del size=831060 2023-12-03T11:30:41.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.299+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=F10WTZFSUVWL type=del 2023-12-03T11:30:41.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.301+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.0 size=1048576 2023-12-03T11:30:41.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.302+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.1 size=1048576 2023-12-03T11:30:41.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.303+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.2 size=659614 2023-12-03T11:30:41.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.303+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=E0W0E7KD0S8B tag=56UZGW9HPK6K type=write size=3805342 2023-12-03T11:30:41.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.305+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=LZ9R1VV1S2J2 type=del 2023-12-03T11:30:41.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.307+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.0 size=1048576 2023-12-03T11:30:41.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.308+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.1 size=1048576 2023-12-03T11:30:41.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.308+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.2 size=288361 2023-12-03T11:30:41.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.308+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=G9RHJHSL2S3G tag=ZOKKCCJ7U9R3 type=write size=3434089 2023-12-03T11:30:41.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.310+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=QZRBB476497T type=del 2023-12-03T11:30:41.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.312+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~NMRCK401HIJB.0 size=98194 2023-12-03T11:30:41.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.312+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=NMRCK401HIJB tag=Q9H8VXAF1KI7 type=write size=1146770 2023-12-03T11:30:41.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.313+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~1BRROK27YMHC.0 size=1048576 2023-12-03T11:30:41.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.314+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~1BRROK27YMHC.1 size=404670 2023-12-03T11:30:41.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.316+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=1BRROK27YMHC tag=4IUF3BP9SRGM type=write size=2501822 2023-12-03T11:30:41.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.317+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=259L13DUB2YF type=write size=436842 2023-12-03T11:30:41.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.318+0000 7fc553ded780 1 < canceled operation key=SWTRHXTAR0A1 tag=S4G455NKTBPQ type=write size=880511 2023-12-03T11:30:41.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.320+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=ZQGCSVRH74Z1 type=del size=1399731 2023-12-03T11:30:41.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.321+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.0 size=1048576 2023-12-03T11:30:41.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.322+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.1 size=1048576 2023-12-03T11:30:41.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.323+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~FNXF9RWG7VNF.2 size=646029 2023-12-03T11:30:41.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.325+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=FNXF9RWG7VNF tag=6O5M302JKE07 type=write size=3791757 2023-12-03T11:30:41.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.326+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=5LA8EB3HSDIE type=del size=881994 2023-12-03T11:30:41.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.327+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=DGDEFO36YSFU type=del 2023-12-03T11:30:41.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.329+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=DXGAAN7WPQRQ type=del size=2887341 2023-12-03T11:30:41.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.333+0000 7fc553ded780 1 < suggested remove operation key=L4ME9RYRR4NG tag=66PW16082187 type=del 2023-12-03T11:30:41.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.334+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=42FQUSL8I7EA type=del size=3873867 2023-12-03T11:30:41.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.336+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~1JU7SI5A9SUL.0 size=956916 2023-12-03T11:30:41.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.337+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=1JU7SI5A9SUL tag=9RRWS7FBPZL4 type=write size=2005492 2023-12-03T11:30:41.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.339+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=FWYJB1G5C9O1 type=del size=771259 2023-12-03T11:30:41.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.340+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=0RJM7F2S7VCN type=del 2023-12-03T11:30:41.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.343+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.0 size=1048576 2023-12-03T11:30:41.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.344+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.1 size=1048576 2023-12-03T11:30:41.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.345+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.2 size=768247 2023-12-03T11:30:41.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.345+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=HF2UZH8UE6QI tag=UAEM472DURH4 type=write size=3913975 2023-12-03T11:30:41.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.345+0000 7fc553ded780 1 < canceled part key=_multipart_WLTMDGRBF4C9.2~WXWY20WKY702.0 size=946797 2023-12-03T11:30:41.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.347+0000 7fc553ded780 1 < canceled multipart upload key=WLTMDGRBF4C9 upload=WXWY20WKY702 tag=UM4PZTF0EXRZ type=write size=1995373 2023-12-03T11:30:41.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.350+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~Z3Y1H214XR32.0 size=1048576 2023-12-03T11:30:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.351+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~Z3Y1H214XR32.1 size=928632 2023-12-03T11:30:41.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.351+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=Z3Y1H214XR32 tag=NYASQX9PU81J type=write size=3025784 2023-12-03T11:30:41.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.352+0000 7fc553ded780 -1 < failed to complete operation key=L4ME9RYRR4NG tag=M0JI2BFD6TX1 type=write size=289508: (22) Invalid argument 2023-12-03T11:30:41.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.355+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.0 size=1048576 2023-12-03T11:30:41.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.356+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.1 size=1048576 2023-12-03T11:30:41.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.357+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.2 size=829128 2023-12-03T11:30:41.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.357+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=Q801LC168ESY tag=US2RNQT2HU0J type=write size=3974856 2023-12-03T11:30:41.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.358+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~20C8LH5FOFSL.0 size=1048576 2023-12-03T11:30:41.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.359+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~20C8LH5FOFSL.1 size=871878 2023-12-03T11:30:41.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.362+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=20C8LH5FOFSL tag=IMW6870G2Z1G type=write size=2969030 2023-12-03T11:30:41.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.363+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=R2MEQSXHVOTQ type=write size=191031 2023-12-03T11:30:41.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.365+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.0 size=1048576 2023-12-03T11:30:41.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.366+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.1 size=1048576 2023-12-03T11:30:41.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.367+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~J7N2XXEBZ94V.2 size=776121 2023-12-03T11:30:41.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.369+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=J7N2XXEBZ94V tag=DN83FUSJNO9F type=write size=3921849 2023-12-03T11:30:41.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.372+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~WMUVS3O8WBKZ.0 size=813450 2023-12-03T11:30:41.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.372+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=WMUVS3O8WBKZ tag=OP4WCUNHIDT3 type=write size=1862026 2023-12-03T11:30:41.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.373+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JCEJA3LV8ZL2.0 size=787450 2023-12-03T11:30:41.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.376+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=JCEJA3LV8ZL2 tag=Z6UKOUJR2DVK type=write size=1836026 2023-12-03T11:30:41.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.378+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~WIK1TZWPWWZP.0 size=1048576 2023-12-03T11:30:41.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.379+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~WIK1TZWPWWZP.1 size=762229 2023-12-03T11:30:41.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.379+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=WIK1TZWPWWZP tag=H5SBAK9RBD60 type=write size=2859381 2023-12-03T11:30:41.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.381+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=DW8QBT2PUKS2 type=del 2023-12-03T11:30:41.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.382+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=7YAIQ8059MZ2 type=del size=2921674 2023-12-03T11:30:41.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.383+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.0 size=1048576 2023-12-03T11:30:41.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.384+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.1 size=1048576 2023-12-03T11:30:41.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.385+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~E0W0E7KD0S8B.2 size=659614 2023-12-03T11:30:41.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.387+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=E0W0E7KD0S8B tag=56UZGW9HPK6K type=write size=3805342 2023-12-03T11:30:41.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.389+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=XE1SF17FRA4N type=del size=550005 2023-12-03T11:30:41.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.390+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.0 size=1048576 2023-12-03T11:30:41.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.391+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.1 size=1048576 2023-12-03T11:30:41.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.392+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~G9RHJHSL2S3G.2 size=288361 2023-12-03T11:30:41.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.394+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=G9RHJHSL2S3G tag=ZOKKCCJ7U9R3 type=write size=3434089 2023-12-03T11:30:41.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.396+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=TEKA9A92LVYS type=del size=2164661 2023-12-03T11:30:41.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.396+0000 7fc553ded780 1 < canceled part key=_multipart_G1OBBD75M5ZQ.2~NMRCK401HIJB.0 size=98194 2023-12-03T11:30:41.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.398+0000 7fc553ded780 1 < canceled multipart upload key=G1OBBD75M5ZQ upload=NMRCK401HIJB tag=Q9H8VXAF1KI7 type=write size=1146770 2023-12-03T11:30:41.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.399+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=RJCKVUUXYVJ3 type=del size=3850377 2023-12-03T11:30:41.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.400+0000 7fc553ded780 1 < completed write operation key=WDV4CW863V33 tag=259L13DUB2YF type=write size=436842 2023-12-03T11:30:41.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.402+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=83O2CNL5SIXG type=del size=3359085 2023-12-03T11:30:41.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.403+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=ZQGCSVRH74Z1 type=del 2023-12-03T11:30:41.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.404+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=Z19Q8CJ75E36 type=del size=3596473 2023-12-03T11:30:41.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.406+0000 7fc553ded780 1 < suggested remove operation key=RO0KGI2QMA71 tag=5LA8EB3HSDIE type=del 2023-12-03T11:30:41.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.407+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=RKMGNPEWI5ZU type=del size=3495045 2023-12-03T11:30:41.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.409+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=DXGAAN7WPQRQ type=del 2023-12-03T11:30:41.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.410+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=PO4T5JVCYU1P type=write size=338576 2023-12-03T11:30:41.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.411+0000 7fc553ded780 1 < canceled operation key=EK7V6U9DSLD0 tag=42FQUSL8I7EA type=del size=3873867 2023-12-03T11:30:41.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.413+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~UF16JL8K9G8B.0 size=1048576 2023-12-03T11:30:41.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.414+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~UF16JL8K9G8B.1 size=398834 2023-12-03T11:30:41.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.414+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=UF16JL8K9G8B tag=R4F2LP9675PV type=write size=2495986 2023-12-03T11:30:41.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.416+0000 7fc553ded780 1 < suggested update operation key=ZUNOFY5N3J3H tag=FWYJB1G5C9O1 type=del size=771259 2023-12-03T11:30:41.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.417+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=YRSR9RGH2G5N type=del size=940560 2023-12-03T11:30:41.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.419+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.0 size=1048576 2023-12-03T11:30:41.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.420+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.1 size=1048576 2023-12-03T11:30:41.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.421+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HF2UZH8UE6QI.2 size=768247 2023-12-03T11:30:41.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.423+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=HF2UZH8UE6QI tag=UAEM472DURH4 type=write size=3913975 2023-12-03T11:30:41.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.424+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=JPRRWATS1TBI type=del size=2805827 2023-12-03T11:30:41.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.425+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~Z3Y1H214XR32.0 size=1048576 2023-12-03T11:30:41.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.426+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~Z3Y1H214XR32.1 size=928632 2023-12-03T11:30:41.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.428+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=Z3Y1H214XR32 tag=NYASQX9PU81J type=write size=3025784 2023-12-03T11:30:41.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.429+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=O92J5EQD06R4 type=del size=3196025 2023-12-03T11:30:41.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.431+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.0 size=1048576 2023-12-03T11:30:41.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.431+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.1 size=1048576 2023-12-03T11:30:41.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.433+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~Q801LC168ESY.2 size=829128 2023-12-03T11:30:41.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.435+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=Q801LC168ESY tag=US2RNQT2HU0J type=write size=3974856 2023-12-03T11:30:41.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.436+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=BIOG3MIYS9OE type=del size=3482403 2023-12-03T11:30:41.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.437+0000 7fc553ded780 1 < canceled operation key=CZP8OQRU641T tag=R2MEQSXHVOTQ type=write size=191031 2023-12-03T11:30:41.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.438+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=CBU7M72DLIPM type=del size=3326557 2023-12-03T11:30:41.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.439+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~WMUVS3O8WBKZ.0 size=813450 2023-12-03T11:30:41.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.441+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=WMUVS3O8WBKZ tag=OP4WCUNHIDT3 type=write size=1862026 2023-12-03T11:30:41.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.442+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=OZ0UU2ZJN0LR type=write size=606583 2023-12-03T11:30:41.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.444+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~WIK1TZWPWWZP.0 size=1048576 2023-12-03T11:30:41.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.445+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~WIK1TZWPWWZP.1 size=762229 2023-12-03T11:30:41.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.447+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=WIK1TZWPWWZP tag=H5SBAK9RBD60 type=write size=2859381 2023-12-03T11:30:41.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.451+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~N27ZO9ITQQ3F.0 size=548466 2023-12-03T11:30:41.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.451+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=N27ZO9ITQQ3F tag=6GNGJY3NKWG2 type=write size=1597042 2023-12-03T11:30:41.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.452+0000 7fc553ded780 1 < canceled operation key=FP2UZMDKYTD1 tag=7YAIQ8059MZ2 type=del size=2921674 2023-12-03T11:30:41.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.453+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=JLWYS8SFAF5D type=del size=3335326 2023-12-03T11:30:41.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.455+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=XE1SF17FRA4N type=del 2023-12-03T11:30:41.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.456+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=LXJ58FLV050Z type=write size=12055 2023-12-03T11:30:41.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.456+0000 7fc553ded780 -1 < no bi entry to suggest for operation key=RO0KGI2QMA71 tag=TEKA9A92LVYS type=del size=2164661: (2) No such file or directory 2023-12-03T11:30:41.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.458+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=6FSLWQ7S2W6B type=del size=920399 2023-12-03T11:30:41.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.459+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=RJCKVUUXYVJ3 type=del 2023-12-03T11:30:41.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.461+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.0 size=1048576 2023-12-03T11:30:41.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.462+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.1 size=1048576 2023-12-03T11:30:41.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.463+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.2 size=483208 2023-12-03T11:30:41.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.463+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=UWLIF14AAQQG tag=2QVR5DD6GC9R type=write size=3628936 2023-12-03T11:30:41.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.465+0000 7fc553ded780 1 < suggested update operation key=YE73HBF64CS7 tag=83O2CNL5SIXG type=del size=3359085 2023-12-03T11:30:41.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.466+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=9SVIID0OI7Y6 type=del size=1152159 2023-12-03T11:30:41.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.468+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=Z19Q8CJ75E36 type=del 2023-12-03T11:30:41.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.469+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=6KA5RP857YLO type=del size=2652894 2023-12-03T11:30:41.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.470+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=RKMGNPEWI5ZU type=del 2023-12-03T11:30:41.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.472+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.0 size=1048576 2023-12-03T11:30:41.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.473+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.1 size=1048576 2023-12-03T11:30:41.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.474+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.2 size=218366 2023-12-03T11:30:41.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.474+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=D3NDTV5U7CFN tag=M2ETUT7OLEWT type=write size=3364094 2023-12-03T11:30:41.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.475+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=PO4T5JVCYU1P type=write size=338576 2023-12-03T11:30:41.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.477+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.0 size=1048576 2023-12-03T11:30:41.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.478+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.1 size=1048576 2023-12-03T11:30:41.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.479+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.2 size=723678 2023-12-03T11:30:41.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.479+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=XFWIG43RKPHO tag=COEIL8C3MUC6 type=write size=3869406 2023-12-03T11:30:41.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.480+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~UF16JL8K9G8B.0 size=1048576 2023-12-03T11:30:41.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.481+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~UF16JL8K9G8B.1 size=398834 2023-12-03T11:30:41.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.483+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=UF16JL8K9G8B tag=R4F2LP9675PV type=write size=2495986 2023-12-03T11:30:41.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.485+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=BQCB9SSJK9L0 type=del size=933540 2023-12-03T11:30:41.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.486+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=YRSR9RGH2G5N type=del 2023-12-03T11:30:41.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.488+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~IDCX2WG7MUTV.0 size=732523 2023-12-03T11:30:41.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.488+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=IDCX2WG7MUTV tag=QC6ALOPUD7LV type=write size=1781099 2023-12-03T11:30:41.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.491+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=JPRRWATS1TBI type=del size=2805827 2023-12-03T11:30:41.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.492+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=2IP61S0LH7VF type=write size=880030 2023-12-03T11:30:41.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.493+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=O92J5EQD06R4 type=del 2023-12-03T11:30:41.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.495+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.0 size=1048576 2023-12-03T11:30:41.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.496+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.1 size=1048576 2023-12-03T11:30:41.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.497+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.2 size=1023787 2023-12-03T11:30:41.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.497+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=S4T8ORZ0DZ45 tag=EQJUTF9FTKQ5 type=write size=4169515 2023-12-03T11:30:41.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.498+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=BIOG3MIYS9OE type=del size=3482403 2023-12-03T11:30:41.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.499+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=8KBFDVMW9OE6 type=del size=3869761 2023-12-03T11:30:41.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.502+0000 7fc553ded780 1 < suggested remove operation key=C2TPF9Q5Z7XQ tag=CBU7M72DLIPM type=del 2023-12-03T11:30:41.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.504+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~S2BYX8L4U8YZ.0 size=605340 2023-12-03T11:30:41.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.504+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=S2BYX8L4U8YZ tag=B19GROWZFNO7 type=write size=1653916 2023-12-03T11:30:41.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.505+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=OZ0UU2ZJN0LR type=write size=606583 2023-12-03T11:30:41.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.506+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=CSEJ61UB53XS type=del size=214877 2023-12-03T11:30:41.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.507+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~N27ZO9ITQQ3F.0 size=548466 2023-12-03T11:30:41.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.509+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=N27ZO9ITQQ3F tag=6GNGJY3NKWG2 type=write size=1597042 2023-12-03T11:30:41.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.511+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=8N6U8UHB9GFS type=del size=1027835 2023-12-03T11:30:41.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.512+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=JLWYS8SFAF5D type=del 2023-12-03T11:30:41.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.514+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~MVF0YLPZ2NFQ.0 size=1048576 2023-12-03T11:30:41.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.515+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~MVF0YLPZ2NFQ.1 size=464058 2023-12-03T11:30:41.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.515+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=MVF0YLPZ2NFQ tag=PBUILWYDSZR9 type=write size=2561210 2023-12-03T11:30:41.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.516+0000 7fc553ded780 1 < completed write operation key=ZHK2M78A0SLQ tag=LXJ58FLV050Z type=write size=12055 2023-12-03T11:30:41.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.518+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=639KQ5623UVF type=del size=368138 2023-12-03T11:30:41.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.520+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=6FSLWQ7S2W6B type=del 2023-12-03T11:30:41.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.521+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=GY3R1LN6TSRV type=del size=2556865 2023-12-03T11:30:41.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.523+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.0 size=1048576 2023-12-03T11:30:41.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.524+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.1 size=1048576 2023-12-03T11:30:41.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.525+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~UWLIF14AAQQG.2 size=483208 2023-12-03T11:30:41.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.527+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=UWLIF14AAQQG tag=2QVR5DD6GC9R type=write size=3628936 2023-12-03T11:30:41.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.528+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=PJTZGUDDKRAK type=del size=1639613 2023-12-03T11:30:41.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.530+0000 7fc553ded780 1 < suggested remove operation key=4TWM1QM1OHOF tag=9SVIID0OI7Y6 type=del 2023-12-03T11:30:41.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.531+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=6S8OSGMQFDM3 type=write size=1029502 2023-12-03T11:30:41.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.534+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=6KA5RP857YLO type=del 2023-12-03T11:30:41.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.536+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~5Z2EIAKUFSV1.0 size=1048576 2023-12-03T11:30:41.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.537+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~5Z2EIAKUFSV1.1 size=469517 2023-12-03T11:30:41.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.537+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=5Z2EIAKUFSV1 tag=XETHWF9PNXVV type=write size=2566669 2023-12-03T11:30:41.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.538+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.0 size=1048576 2023-12-03T11:30:41.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.539+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.1 size=1048576 2023-12-03T11:30:41.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.540+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~D3NDTV5U7CFN.2 size=218366 2023-12-03T11:30:41.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.542+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=D3NDTV5U7CFN tag=M2ETUT7OLEWT type=write size=3364094 2023-12-03T11:30:41.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.543+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=H6R0TQ12DBIR type=del size=2021068 2023-12-03T11:30:41.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.544+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.0 size=1048576 2023-12-03T11:30:41.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.547+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.1 size=1048576 2023-12-03T11:30:41.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.548+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~XFWIG43RKPHO.2 size=723678 2023-12-03T11:30:41.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.550+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=XFWIG43RKPHO tag=COEIL8C3MUC6 type=write size=3869406 2023-12-03T11:30:41.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.551+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=CLZVS7HP72G9 type=del size=792685 2023-12-03T11:30:41.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.554+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=BQCB9SSJK9L0 type=del size=933540 2023-12-03T11:30:41.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.555+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=AF9L51L3VEAR type=del size=249785 2023-12-03T11:30:41.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.556+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~IDCX2WG7MUTV.0 size=732523 2023-12-03T11:30:41.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.558+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=IDCX2WG7MUTV tag=QC6ALOPUD7LV type=write size=1781099 2023-12-03T11:30:41.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.560+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~LD81VMVTXVKB.0 size=1048576 2023-12-03T11:30:41.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.561+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~LD81VMVTXVKB.1 size=602666 2023-12-03T11:30:41.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.561+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=LD81VMVTXVKB tag=3FEN5ISOTALW type=write size=2699818 2023-12-03T11:30:41.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.563+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=2IP61S0LH7VF type=write size=880030 2023-12-03T11:30:41.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.566+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~88TWMPDM2WUN.0 size=1048576 2023-12-03T11:30:41.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.568+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~88TWMPDM2WUN.1 size=913994 2023-12-03T11:30:41.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.568+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=88TWMPDM2WUN tag=FIFVOFQY2EB4 type=write size=3011146 2023-12-03T11:30:41.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.569+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.0 size=1048576 2023-12-03T11:30:41.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.570+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.1 size=1048576 2023-12-03T11:30:41.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.571+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S4T8ORZ0DZ45.2 size=1023787 2023-12-03T11:30:41.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.573+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=S4T8ORZ0DZ45 tag=EQJUTF9FTKQ5 type=write size=4169515 2023-12-03T11:30:41.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.574+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=WJUUJV2DH748 type=del size=1718829 2023-12-03T11:30:41.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.575+0000 7fc553ded780 -1 < failed to complete operation key=JUDXD9DL4N43 tag=8KBFDVMW9OE6 type=del size=3869761: (22) Invalid argument 2023-12-03T11:30:41.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.577+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.0 size=1048576 2023-12-03T11:30:41.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.578+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.1 size=1048576 2023-12-03T11:30:41.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.579+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.2 size=30947 2023-12-03T11:30:41.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.579+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=47TA66MXKLTY tag=QSM48I1EOQMA type=write size=3176675 2023-12-03T11:30:41.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.580+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~S2BYX8L4U8YZ.0 size=605340 2023-12-03T11:30:41.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.582+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=S2BYX8L4U8YZ tag=B19GROWZFNO7 type=write size=1653916 2023-12-03T11:30:41.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.584+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=AI3BNU5KFO0A type=del size=2024911 2023-12-03T11:30:41.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.586+0000 7fc553ded780 1 < suggested update operation key=G1OBBD75M5ZQ tag=CSEJ61UB53XS type=del size=214877 2023-12-03T11:30:41.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.587+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=OZFCHWSOCUOQ type=del size=2572458 2023-12-03T11:30:41.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.588+0000 7fc553ded780 -1 < failed to complete operation key=0H5OZMBMLQFJ tag=8N6U8UHB9GFS type=del size=1027835: (22) Invalid argument 2023-12-03T11:30:41.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.591+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~V3QANBDRKQI3.0 size=1048576 2023-12-03T11:30:41.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.591+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~V3QANBDRKQI3.1 size=874133 2023-12-03T11:30:41.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.591+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=V3QANBDRKQI3 tag=LCGG8BIM7YUD type=write size=2971285 2023-12-03T11:30:41.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.593+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~MVF0YLPZ2NFQ.0 size=1048576 2023-12-03T11:30:41.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.594+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~MVF0YLPZ2NFQ.1 size=464058 2023-12-03T11:30:41.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.596+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=MVF0YLPZ2NFQ tag=PBUILWYDSZR9 type=write size=2561210 2023-12-03T11:30:41.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.597+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=YH7JIJGCKJ07 type=del size=546542 2023-12-03T11:30:41.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.598+0000 7fc553ded780 -1 < failed to complete operation key=JUDXD9DL4N43 tag=639KQ5623UVF type=del size=368138: (22) Invalid argument 2023-12-03T11:30:41.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.600+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.0 size=1048576 2023-12-03T11:30:41.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.601+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.1 size=1048576 2023-12-03T11:30:41.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.602+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.2 size=662396 2023-12-03T11:30:41.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.602+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=69XQ53QN1YHO tag=X50Q7AGBCEAJ type=write size=3808124 2023-12-03T11:30:41.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.604+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=GY3R1LN6TSRV type=del 2023-12-03T11:30:41.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.606+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.0 size=1048576 2023-12-03T11:30:41.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.607+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.1 size=1048576 2023-12-03T11:30:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.608+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.2 size=873569 2023-12-03T11:30:41.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.608+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=V3R27CQI5C71 tag=M2X35HPQPLKD type=write size=4019297 2023-12-03T11:30:41.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.611+0000 7fc553ded780 1 < suggested remove operation key=YE73HBF64CS7 tag=PJTZGUDDKRAK type=del 2023-12-03T11:30:41.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.612+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=Y2M9X5GA9WOS type=del size=1263438 2023-12-03T11:30:41.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.613+0000 7fc553ded780 1 < completed write operation key=J664UUE697Q8 tag=6S8OSGMQFDM3 type=write size=1029502 2023-12-03T11:30:41.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.615+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=T6RY0YI3CRVP type=del size=4147001 2023-12-03T11:30:41.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.616+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~5Z2EIAKUFSV1.0 size=1048576 2023-12-03T11:30:41.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.617+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~5Z2EIAKUFSV1.1 size=469517 2023-12-03T11:30:41.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.620+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=5Z2EIAKUFSV1 tag=XETHWF9PNXVV type=write size=2566669 2023-12-03T11:30:41.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.622+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~FYACCP9Y0YQM.0 size=1048576 2023-12-03T11:30:41.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.623+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~FYACCP9Y0YQM.1 size=411379 2023-12-03T11:30:41.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.623+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=FYACCP9Y0YQM tag=VIZ9LD03YOY2 type=write size=2508531 2023-12-03T11:30:41.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.625+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=H6R0TQ12DBIR type=del 2023-12-03T11:30:41.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.626+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=ANMRS69PIQG2 type=del size=593398 2023-12-03T11:30:41.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.628+0000 7fc553ded780 1 < suggested update operation key=J664UUE697Q8 tag=CLZVS7HP72G9 type=del size=792685 2023-12-03T11:30:41.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.630+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.0 size=1048576 2023-12-03T11:30:41.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.632+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.1 size=1048576 2023-12-03T11:30:41.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.633+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.2 size=331949 2023-12-03T11:30:41.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.633+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=0HVCQ9PS8VBY tag=FUDSSNDZ4GHP type=write size=3477677 2023-12-03T11:30:41.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.633+0000 7fc553ded780 -1 < no bi entry to suggest for operation key=YE73HBF64CS7 tag=AF9L51L3VEAR type=del size=249785: (2) No such file or directory 2023-12-03T11:30:41.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.635+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~7ZXVDPDXALS7.0 size=160048 2023-12-03T11:30:41.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.635+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=7ZXVDPDXALS7 tag=CY9X5M0WO7JD type=write size=1208624 2023-12-03T11:30:41.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.637+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~LD81VMVTXVKB.0 size=1048576 2023-12-03T11:30:41.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.639+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~LD81VMVTXVKB.1 size=602666 2023-12-03T11:30:41.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.641+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=LD81VMVTXVKB tag=3FEN5ISOTALW type=write size=2699818 2023-12-03T11:30:41.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.643+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~8YY78B2AK08X.0 size=1048576 2023-12-03T11:30:41.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.644+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~8YY78B2AK08X.1 size=615012 2023-12-03T11:30:41.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.644+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=8YY78B2AK08X tag=AEPUXJOOPYYT type=write size=2712164 2023-12-03T11:30:41.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.646+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~88TWMPDM2WUN.0 size=1048576 2023-12-03T11:30:41.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.647+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~88TWMPDM2WUN.1 size=913994 2023-12-03T11:30:41.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.649+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=88TWMPDM2WUN tag=FIFVOFQY2EB4 type=write size=3011146 2023-12-03T11:30:41.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.651+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=70XW2NTDBDX9 type=del size=1732336 2023-12-03T11:30:41.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.652+0000 7fc553ded780 1 < canceled operation key=C2TPF9Q5Z7XQ tag=WJUUJV2DH748 type=del size=1718829 2023-12-03T11:30:41.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.653+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=TWOLVP1K4QT4 type=del size=1404960 2023-12-03T11:30:41.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.654+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.0 size=1048576 2023-12-03T11:30:41.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.656+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.1 size=1048576 2023-12-03T11:30:41.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.657+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~47TA66MXKLTY.2 size=30947 2023-12-03T11:30:41.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.659+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=47TA66MXKLTY tag=QSM48I1EOQMA type=write size=3176675 2023-12-03T11:30:41.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.661+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~KNAQBY0FWIGB.0 size=60575 2023-12-03T11:30:41.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.661+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=KNAQBY0FWIGB tag=C7WCXPFJOX6B type=write size=1109151 2023-12-03T11:30:41.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.663+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=AI3BNU5KFO0A type=del 2023-12-03T11:30:41.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.664+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=O63SR3LR1IYK type=del size=2516527 2023-12-03T11:30:41.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.667+0000 7fc553ded780 1 < suggested remove operation key=0H5OZMBMLQFJ tag=OZFCHWSOCUOQ type=del 2023-12-03T11:30:41.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.668+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=8LDCE24GABQ2 type=del size=4146211 2023-12-03T11:30:41.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.669+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~V3QANBDRKQI3.0 size=1048576 2023-12-03T11:30:41.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.671+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~V3QANBDRKQI3.1 size=874133 2023-12-03T11:30:41.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.673+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=V3QANBDRKQI3 tag=LCGG8BIM7YUD type=write size=2971285 2023-12-03T11:30:41.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.675+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~KYJSR6RFX6GK.0 size=1048576 2023-12-03T11:30:41.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.676+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~KYJSR6RFX6GK.1 size=206175 2023-12-03T11:30:41.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.676+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=KYJSR6RFX6GK tag=OIJGUUO3WUNB type=write size=2303327 2023-12-03T11:30:41.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.677+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=YH7JIJGCKJ07 type=del 2023-12-03T11:30:41.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.680+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I8MVO002Q9LF.0 size=160502 2023-12-03T11:30:41.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.680+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=I8MVO002Q9LF tag=ROVEMI53GCQN type=write size=1209078 2023-12-03T11:30:41.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.681+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.0 size=1048576 2023-12-03T11:30:41.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.683+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.1 size=1048576 2023-12-03T11:30:41.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.684+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~69XQ53QN1YHO.2 size=662396 2023-12-03T11:30:41.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.686+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=69XQ53QN1YHO tag=X50Q7AGBCEAJ type=write size=3808124 2023-12-03T11:30:41.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.689+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~PSR932V55S1O.0 size=822163 2023-12-03T11:30:41.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.689+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=PSR932V55S1O tag=8OY18NVGNC1E type=write size=1870739 2023-12-03T11:30:41.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.690+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.0 size=1048576 2023-12-03T11:30:41.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.691+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.1 size=1048576 2023-12-03T11:30:41.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.692+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~V3R27CQI5C71.2 size=873569 2023-12-03T11:30:41.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.694+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=V3R27CQI5C71 tag=M2X35HPQPLKD type=write size=4019297 2023-12-03T11:30:41.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.695+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=3XCBST6P5LRG type=del size=2650602 2023-12-03T11:30:41.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.697+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=Y2M9X5GA9WOS type=del 2023-12-03T11:30:41.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.698+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=IFX1QSIZETWI type=write size=410675 2023-12-03T11:30:41.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.699+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=T6RY0YI3CRVP type=del 2023-12-03T11:30:41.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.700+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=AHNX3GUVR9RP type=del size=2989824 2023-12-03T11:30:41.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.701+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~FYACCP9Y0YQM.0 size=1048576 2023-12-03T11:30:41.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.703+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~FYACCP9Y0YQM.1 size=411379 2023-12-03T11:30:41.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.705+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=FYACCP9Y0YQM tag=VIZ9LD03YOY2 type=write size=2508531 2023-12-03T11:30:41.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.707+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.0 size=1048576 2023-12-03T11:30:41.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.709+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.1 size=1048576 2023-12-03T11:30:41.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.710+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.2 size=1047669 2023-12-03T11:30:41.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.710+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=D8NDD46CBKJ9 tag=Z76U3DS6R553 type=write size=4193397 2023-12-03T11:30:41.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.711+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=ANMRS69PIQG2 type=del 2023-12-03T11:30:41.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.713+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~J6CXNHP8NGET.0 size=723739 2023-12-03T11:30:41.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.713+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=J6CXNHP8NGET tag=VE4782WE9CPU type=write size=1772315 2023-12-03T11:30:41.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.715+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.0 size=1048576 2023-12-03T11:30:41.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.716+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.1 size=1048576 2023-12-03T11:30:41.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.717+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~0HVCQ9PS8VBY.2 size=331949 2023-12-03T11:30:41.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.720+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=0HVCQ9PS8VBY tag=FUDSSNDZ4GHP type=write size=3477677 2023-12-03T11:30:41.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.722+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.0 size=1048576 2023-12-03T11:30:41.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.723+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.1 size=1048576 2023-12-03T11:30:41.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.724+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.2 size=718276 2023-12-03T11:30:41.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.724+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=BGY21OCM7D6D tag=V93B41PLTNLY type=write size=3864004 2023-12-03T11:30:41.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.726+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~7ZXVDPDXALS7.0 size=160048 2023-12-03T11:30:41.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.728+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=7ZXVDPDXALS7 tag=CY9X5M0WO7JD type=write size=1208624 2023-12-03T11:30:41.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.730+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=SGLCZIDOA2ON type=del size=248133 2023-12-03T11:30:41.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.731+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~8YY78B2AK08X.0 size=1048576 2023-12-03T11:30:41.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.732+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~8YY78B2AK08X.1 size=615012 2023-12-03T11:30:41.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.735+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=8YY78B2AK08X tag=AEPUXJOOPYYT type=write size=2712164 2023-12-03T11:30:41.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.736+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=SA1GGL711ASH type=del size=3366401 2023-12-03T11:30:41.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.739+0000 7fc553ded780 1 < suggested update operation key=JUDXD9DL4N43 tag=70XW2NTDBDX9 type=del size=1732336 2023-12-03T11:30:41.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.740+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=5BHHBZKGHQTH type=del size=3053800 2023-12-03T11:30:41.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.742+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=TWOLVP1K4QT4 type=del 2023-12-03T11:30:41.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.743+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=6SNLV3B7MVVE type=del size=1870099 2023-12-03T11:30:41.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.743+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~KNAQBY0FWIGB.0 size=60575 2023-12-03T11:30:41.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.745+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=KNAQBY0FWIGB tag=C7WCXPFJOX6B type=write size=1109151 2023-12-03T11:30:41.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.747+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=SUYO5TY2AY65 type=del size=1115489 2023-12-03T11:30:41.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.749+0000 7fc553ded780 1 < suggested update operation key=RO0KGI2QMA71 tag=O63SR3LR1IYK type=del size=2516527 2023-12-03T11:30:41.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.750+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=AVW3TIV9NS84 type=del size=3224469 2023-12-03T11:30:41.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.751+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=8LDCE24GABQ2 type=del 2023-12-03T11:30:41.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.752+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=E32NATFY2SEZ type=del size=1792960 2023-12-03T11:30:41.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.754+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~KYJSR6RFX6GK.0 size=1048576 2023-12-03T11:30:41.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.755+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~KYJSR6RFX6GK.1 size=206175 2023-12-03T11:30:41.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.757+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=KYJSR6RFX6GK tag=OIJGUUO3WUNB type=write size=2303327 2023-12-03T11:30:41.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.759+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5ZGS03F2SQOY.0 size=1048576 2023-12-03T11:30:41.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.760+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5ZGS03F2SQOY.1 size=774735 2023-12-03T11:30:41.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.760+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=5ZGS03F2SQOY tag=467C81HRD23Z type=write size=2871887 2023-12-03T11:30:41.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.761+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I8MVO002Q9LF.0 size=160502 2023-12-03T11:30:41.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.763+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=I8MVO002Q9LF tag=ROVEMI53GCQN type=write size=1209078 2023-12-03T11:30:41.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.765+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=YYC5I98VYGGZ type=del size=1560596 2023-12-03T11:30:41.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.766+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~PSR932V55S1O.0 size=822163 2023-12-03T11:30:41.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.768+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=PSR932V55S1O tag=8OY18NVGNC1E type=write size=1870739 2023-12-03T11:30:41.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.770+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.0 size=1048576 2023-12-03T11:30:41.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.771+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.1 size=1048576 2023-12-03T11:30:41.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.772+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.2 size=365679 2023-12-03T11:30:41.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.772+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=W6S95GL3C5CV tag=W1OW34XETXOX type=write size=3511407 2023-12-03T11:30:41.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.773+0000 7fc553ded780 1 < canceled operation key=WOKG1DISIKK0 tag=3XCBST6P5LRG type=del size=2650602 2023-12-03T11:30:41.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.774+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=NHUTQ8SFPGPD type=del size=1351968 2023-12-03T11:30:41.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.775+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=IFX1QSIZETWI type=write size=410675 2023-12-03T11:30:41.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.777+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=FY25HRFZ34H5 type=del size=3208728 2023-12-03T11:30:41.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.778+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=AHNX3GUVR9RP type=del 2023-12-03T11:30:41.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.779+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=TZX7UY4AVAV8 type=del size=3064901 2023-12-03T11:30:41.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.780+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.0 size=1048576 2023-12-03T11:30:41.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.781+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.1 size=1048576 2023-12-03T11:30:41.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.782+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~D8NDD46CBKJ9.2 size=1047669 2023-12-03T11:30:41.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.784+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=D8NDD46CBKJ9 tag=Z76U3DS6R553 type=write size=4193397 2023-12-03T11:30:41.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.786+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.0 size=1048576 2023-12-03T11:30:41.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.787+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.1 size=1048576 2023-12-03T11:30:41.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.788+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.2 size=643142 2023-12-03T11:30:41.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.788+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=FANN7XN8AZNH tag=3YEL61BV4OOY type=write size=3788870 2023-12-03T11:30:41.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.789+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~J6CXNHP8NGET.0 size=723739 2023-12-03T11:30:41.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.791+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=J6CXNHP8NGET tag=VE4782WE9CPU type=write size=1772315 2023-12-03T11:30:41.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.793+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~XN3FQ1MY698O.0 size=788741 2023-12-03T11:30:41.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.793+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=XN3FQ1MY698O tag=0OW1NUASICUK type=write size=1837317 2023-12-03T11:30:41.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.794+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.0 size=1048576 2023-12-03T11:30:41.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.794+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.1 size=1048576 2023-12-03T11:30:41.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.795+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~BGY21OCM7D6D.2 size=718276 2023-12-03T11:30:41.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.797+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=BGY21OCM7D6D tag=V93B41PLTNLY type=write size=3864004 2023-12-03T11:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.800+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~LXQUEAZVW3VE.0 size=781293 2023-12-03T11:30:41.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.800+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=LXQUEAZVW3VE tag=AW3EGXWNM8RP type=write size=1829869 2023-12-03T11:30:41.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.802+0000 7fc553ded780 1 < suggested update operation key=WLTMDGRBF4C9 tag=SGLCZIDOA2ON type=del size=248133 2023-12-03T11:30:41.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.805+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~COHC5DVLTC3Z.0 size=1048576 2023-12-03T11:30:41.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.806+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~COHC5DVLTC3Z.1 size=959632 2023-12-03T11:30:41.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.806+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=COHC5DVLTC3Z tag=6TUW1BE1H4KF type=write size=3056784 2023-12-03T11:30:41.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.807+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=SA1GGL711ASH type=del 2023-12-03T11:30:41.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.809+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.0 size=1048576 2023-12-03T11:30:41.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.810+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.1 size=1048576 2023-12-03T11:30:41.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.811+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.2 size=906081 2023-12-03T11:30:41.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.811+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=MXYFW0MPI05Y tag=HOXA78GCPRW0 type=write size=4051809 2023-12-03T11:30:41.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.813+0000 7fc553ded780 1 < suggested remove operation key=UKAL8FCVIRSH tag=5BHHBZKGHQTH type=del 2023-12-03T11:30:41.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.815+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~IHKQN08QQRGZ.0 size=230793 2023-12-03T11:30:41.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.815+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=IHKQN08QQRGZ tag=R04BOTKMNNH2 type=write size=1279369 2023-12-03T11:30:41.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.816+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=6SNLV3B7MVVE type=del 2023-12-03T11:30:41.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.818+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.0 size=1048576 2023-12-03T11:30:41.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.819+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.1 size=1048576 2023-12-03T11:30:41.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.820+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.2 size=410582 2023-12-03T11:30:41.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.820+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=TLAQOLG3TMKD tag=JT8I1A69AIIT type=write size=3556310 2023-12-03T11:30:41.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.821+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=SUYO5TY2AY65 type=del 2023-12-03T11:30:41.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.822+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=0DLGS9KWE92L type=write size=784156 2023-12-03T11:30:41.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.824+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=AVW3TIV9NS84 type=del size=3224469 2023-12-03T11:30:41.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.825+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=V9KL9QKMIAGH type=del size=1461377 2023-12-03T11:30:41.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.826+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=E32NATFY2SEZ type=del 2023-12-03T11:30:41.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.827+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=P240QI7GVIAQ type=del size=3478409 2023-12-03T11:30:41.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.828+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~5ZGS03F2SQOY.0 size=1048576 2023-12-03T11:30:41.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.829+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~5ZGS03F2SQOY.1 size=774735 2023-12-03T11:30:41.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.831+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=5ZGS03F2SQOY tag=467C81HRD23Z type=write size=2871887 2023-12-03T11:30:41.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.833+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=EYTSPNNP4GXH type=del size=1608275 2023-12-03T11:30:41.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.834+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=YYC5I98VYGGZ type=del 2023-12-03T11:30:41.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.836+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.0 size=1048576 2023-12-03T11:30:41.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.837+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.1 size=1048576 2023-12-03T11:30:41.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.838+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.2 size=723277 2023-12-03T11:30:41.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.838+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=FJXBJI1E3RY4 tag=Q54LM6QNP23T type=write size=3869005 2023-12-03T11:30:41.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.839+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.0 size=1048576 2023-12-03T11:30:41.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.840+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.1 size=1048576 2023-12-03T11:30:41.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.841+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~W6S95GL3C5CV.2 size=365679 2023-12-03T11:30:41.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.843+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=W6S95GL3C5CV tag=W1OW34XETXOX type=write size=3511407 2023-12-03T11:30:41.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.846+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~PSP5FFN4TY15.0 size=354298 2023-12-03T11:30:41.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.846+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=PSP5FFN4TY15 tag=FK6HKZVNUVZH type=write size=1402874 2023-12-03T11:30:41.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.847+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=NHUTQ8SFPGPD type=del 2023-12-03T11:30:41.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.848+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=LC3NCCC2W0UT type=write size=1039811 2023-12-03T11:30:41.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.850+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=FY25HRFZ34H5 type=del 2023-12-03T11:30:41.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.852+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~RSI9ZOFVSE14.0 size=96015 2023-12-03T11:30:41.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.852+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=RSI9ZOFVSE14 tag=HNKCP6WSPOKG type=write size=1144591 2023-12-03T11:30:41.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.853+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=TZX7UY4AVAV8 type=del 2023-12-03T11:30:41.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.855+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~BUKVSZ2FYFRV.0 size=1048576 2023-12-03T11:30:41.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.856+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~BUKVSZ2FYFRV.1 size=392932 2023-12-03T11:30:41.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.856+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=BUKVSZ2FYFRV tag=ZEJFMJU72XPT type=write size=2490084 2023-12-03T11:30:41.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.857+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.0 size=1048576 2023-12-03T11:30:41.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.858+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.1 size=1048576 2023-12-03T11:30:41.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.859+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~FANN7XN8AZNH.2 size=643142 2023-12-03T11:30:41.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.861+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=FANN7XN8AZNH tag=3YEL61BV4OOY type=write size=3788870 2023-12-03T11:30:41.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.862+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=NF1NGID4DKPP type=del size=3469002 2023-12-03T11:30:41.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.864+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~XN3FQ1MY698O.0 size=788741 2023-12-03T11:30:41.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.866+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=XN3FQ1MY698O tag=0OW1NUASICUK type=write size=1837317 2023-12-03T11:30:41.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.870+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.0 size=1048576 2023-12-03T11:30:41.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.871+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.1 size=1048576 2023-12-03T11:30:41.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.872+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.2 size=129668 2023-12-03T11:30:41.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.872+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=SAIKEZ0TK023 tag=LYHLC192SRFL type=write size=3275396 2023-12-03T11:30:41.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.873+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~LXQUEAZVW3VE.0 size=781293 2023-12-03T11:30:41.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.875+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=LXQUEAZVW3VE tag=AW3EGXWNM8RP type=write size=1829869 2023-12-03T11:30:41.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.876+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=1YJLKB6QEF84 type=write size=320493 2023-12-03T11:30:41.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.877+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~COHC5DVLTC3Z.0 size=1048576 2023-12-03T11:30:41.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.878+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~COHC5DVLTC3Z.1 size=959632 2023-12-03T11:30:41.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.880+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=COHC5DVLTC3Z tag=6TUW1BE1H4KF type=write size=3056784 2023-12-03T11:30:41.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.882+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=JJY3IQ2K4DOH type=write size=33044 2023-12-03T11:30:41.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.883+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.0 size=1048576 2023-12-03T11:30:41.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.884+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.1 size=1048576 2023-12-03T11:30:41.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.885+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~MXYFW0MPI05Y.2 size=906081 2023-12-03T11:30:41.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.887+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=MXYFW0MPI05Y tag=HOXA78GCPRW0 type=write size=4051809 2023-12-03T11:30:41.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.888+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=87ZOQ6XRF8IF type=del size=1880271 2023-12-03T11:30:41.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.889+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~IHKQN08QQRGZ.0 size=230793 2023-12-03T11:30:41.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.891+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=IHKQN08QQRGZ tag=R04BOTKMNNH2 type=write size=1279369 2023-12-03T11:30:41.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.893+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~A2JU8XMLW2KO.0 size=515568 2023-12-03T11:30:41.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.893+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=A2JU8XMLW2KO tag=AQ2RL1OQIFSZ type=write size=1564144 2023-12-03T11:30:41.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.894+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.0 size=1048576 2023-12-03T11:30:41.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.894+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.1 size=1048576 2023-12-03T11:30:41.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.896+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~TLAQOLG3TMKD.2 size=410582 2023-12-03T11:30:41.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.898+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=TLAQOLG3TMKD tag=JT8I1A69AIIT type=write size=3556310 2023-12-03T11:30:41.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.899+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=I7LQOC09THPU type=del size=317774 2023-12-03T11:30:41.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.900+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=0DLGS9KWE92L type=write size=784156 2023-12-03T11:30:41.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.903+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.0 size=1048576 2023-12-03T11:30:41.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.904+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.1 size=1048576 2023-12-03T11:30:41.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.905+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.2 size=48221 2023-12-03T11:30:41.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.905+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=0PHRWBNA15MD tag=Z02R5H4ITWUL type=write size=3193949 2023-12-03T11:30:41.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.907+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=V9KL9QKMIAGH type=del 2023-12-03T11:30:41.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.908+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=PBJ0R0SMHLHF type=del size=3859072 2023-12-03T11:30:41.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.909+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=P240QI7GVIAQ type=del 2023-12-03T11:30:41.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.912+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~U9JRC8UOV3GN.0 size=1048576 2023-12-03T11:30:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.913+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~U9JRC8UOV3GN.1 size=672738 2023-12-03T11:30:41.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.913+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=U9JRC8UOV3GN tag=99B3N72RAUE0 type=write size=2769890 2023-12-03T11:30:41.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.914+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=EYTSPNNP4GXH type=del 2023-12-03T11:30:41.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.915+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=HUMX59GXSQLE type=del size=1505229 2023-12-03T11:30:41.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.917+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.0 size=1048576 2023-12-03T11:30:41.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.918+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.1 size=1048576 2023-12-03T11:30:41.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.919+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~FJXBJI1E3RY4.2 size=723277 2023-12-03T11:30:41.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.921+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=FJXBJI1E3RY4 tag=Q54LM6QNP23T type=write size=3869005 2023-12-03T11:30:41.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.922+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=H467H3BKEQYB type=del size=4165730 2023-12-03T11:30:41.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.923+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~PSP5FFN4TY15.0 size=354298 2023-12-03T11:30:41.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.925+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=PSP5FFN4TY15 tag=FK6HKZVNUVZH type=write size=1402874 2023-12-03T11:30:41.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.926+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=5QD6G7F1DLE6 type=del size=4174989 2023-12-03T11:30:41.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.928+0000 7fc553ded780 1 < completed write operation key=FP2UZMDKYTD1 tag=LC3NCCC2W0UT type=write size=1039811 2023-12-03T11:30:41.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.930+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~DD9NNNJ4627J.0 size=1949 2023-12-03T11:30:41.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.930+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=DD9NNNJ4627J tag=T838AVIUNYKH type=write size=1050525 2023-12-03T11:30:41.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.931+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~RSI9ZOFVSE14.0 size=96015 2023-12-03T11:30:41.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.933+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=RSI9ZOFVSE14 tag=HNKCP6WSPOKG type=write size=1144591 2023-12-03T11:30:41.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.934+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=54ZMWUGACFYC type=del size=1515503 2023-12-03T11:30:41.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.935+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~BUKVSZ2FYFRV.0 size=1048576 2023-12-03T11:30:41.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.936+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~BUKVSZ2FYFRV.1 size=392932 2023-12-03T11:30:41.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.938+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=BUKVSZ2FYFRV tag=ZEJFMJU72XPT type=write size=2490084 2023-12-03T11:30:41.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.940+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~035NM3OQD7JC.0 size=1048576 2023-12-03T11:30:41.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.941+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~035NM3OQD7JC.1 size=686071 2023-12-03T11:30:41.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.941+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=035NM3OQD7JC tag=PQ5G2Q1M2LEW type=write size=2783223 2023-12-03T11:30:41.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.942+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=NF1NGID4DKPP type=del 2023-12-03T11:30:41.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.945+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.0 size=1048576 2023-12-03T11:30:41.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.946+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.1 size=1048576 2023-12-03T11:30:41.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.947+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.2 size=712122 2023-12-03T11:30:41.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.947+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=M1RHP3R5N9VZ tag=1HIATFUFVSL9 type=write size=3857850 2023-12-03T11:30:41.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.948+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.0 size=1048576 2023-12-03T11:30:41.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.949+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.1 size=1048576 2023-12-03T11:30:41.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.950+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SAIKEZ0TK023.2 size=129668 2023-12-03T11:30:41.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.952+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=SAIKEZ0TK023 tag=LYHLC192SRFL type=write size=3275396 2023-12-03T11:30:41.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.954+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~PR3CF28IAO67.0 size=1048576 2023-12-03T11:30:41.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.955+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~PR3CF28IAO67.1 size=207077 2023-12-03T11:30:41.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.955+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=PR3CF28IAO67 tag=8R23QKXAVL9Z type=write size=2304229 2023-12-03T11:30:41.958 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.957+0000 7fc553ded780 1 < suggested remove operation key=J664UUE697Q8 tag=1YJLKB6QEF84 type=write 2023-12-03T11:30:41.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.958+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=70Q1PBDRWM92 type=del size=2616358 2023-12-03T11:30:41.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.959+0000 7fc553ded780 1 < completed write operation key=WLTMDGRBF4C9 tag=JJY3IQ2K4DOH type=write size=33044 2023-12-03T11:30:41.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.961+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=G5T8SMUV5OYM type=write size=295765 2023-12-03T11:30:41.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.962+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=87ZOQ6XRF8IF type=del 2023-12-03T11:30:41.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.963+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=WQCGFXYHVXZZ type=del size=996330 2023-12-03T11:30:41.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.965+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~A2JU8XMLW2KO.0 size=515568 2023-12-03T11:30:41.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.967+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=A2JU8XMLW2KO tag=AQ2RL1OQIFSZ type=write size=1564144 2023-12-03T11:30:41.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.968+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=YMKCCA0ZWUSX type=del size=1420835 2023-12-03T11:30:41.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.969+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=I7LQOC09THPU type=del 2023-12-03T11:30:41.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.970+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=69334SUJF2DV type=write size=760776 2023-12-03T11:30:41.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.971+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.0 size=1048576 2023-12-03T11:30:41.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.972+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.1 size=1048576 2023-12-03T11:30:41.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.973+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0PHRWBNA15MD.2 size=48221 2023-12-03T11:30:41.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.975+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=0PHRWBNA15MD tag=Z02R5H4ITWUL type=write size=3193949 2023-12-03T11:30:41.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.976+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=DMNFZ9JINVFM type=del size=2860714 2023-12-03T11:30:41.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.977+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=PBJ0R0SMHLHF type=del 2023-12-03T11:30:41.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.979+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=7EJ00NWLKK4H type=write size=569901 2023-12-03T11:30:41.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.980+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~U9JRC8UOV3GN.0 size=1048576 2023-12-03T11:30:41.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.981+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~U9JRC8UOV3GN.1 size=672738 2023-12-03T11:30:41.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.983+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=U9JRC8UOV3GN tag=99B3N72RAUE0 type=write size=2769890 2023-12-03T11:30:41.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.986+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.0 size=1048576 2023-12-03T11:30:41.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.987+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.1 size=1048576 2023-12-03T11:30:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.988+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.2 size=797945 2023-12-03T11:30:41.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.988+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=MTBQ3EETH2I8 tag=NBF8N71WJ1MU type=write size=3943673 2023-12-03T11:30:41.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.990+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=HUMX59GXSQLE type=del 2023-12-03T11:30:41.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.991+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=0GGL3A6F9YSG type=write size=1014391 2023-12-03T11:30:41.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.993+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=H467H3BKEQYB type=del 2023-12-03T11:30:41.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.994+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=5P2ZYTIB51CR type=write size=538076 2023-12-03T11:30:41.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.995+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=5QD6G7F1DLE6 type=del 2023-12-03T11:30:41.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.996+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=ATI8RGZPKCJZ type=write size=559761 2023-12-03T11:30:41.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:41.998+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~DD9NNNJ4627J.0 size=1949 2023-12-03T11:30:42.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.000+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=DD9NNNJ4627J tag=T838AVIUNYKH type=write size=1050525 2023-12-03T11:30:42.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.002+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~SROTFP3G2REH.0 size=1048576 2023-12-03T11:30:42.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.003+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~SROTFP3G2REH.1 size=580404 2023-12-03T11:30:42.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.003+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=SROTFP3G2REH tag=2Z7GRUCUEJPH type=write size=2677556 2023-12-03T11:30:42.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.005+0000 7fc553ded780 1 < suggested remove operation key=FP2UZMDKYTD1 tag=54ZMWUGACFYC type=del 2023-12-03T11:30:42.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.007+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=00SF054AJOGN type=del size=3214356 2023-12-03T11:30:42.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.007+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~035NM3OQD7JC.0 size=1048576 2023-12-03T11:30:42.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.007+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~035NM3OQD7JC.1 size=686071 2023-12-03T11:30:42.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.009+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=035NM3OQD7JC tag=PQ5G2Q1M2LEW type=write size=2783223 2023-12-03T11:30:42.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.011+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~J05J0RXGREIJ.0 size=1048576 2023-12-03T11:30:42.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.012+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~J05J0RXGREIJ.1 size=38127 2023-12-03T11:30:42.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.012+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=J05J0RXGREIJ tag=G7KJ3K97DCVU type=write size=2135279 2023-12-03T11:30:42.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.013+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.0 size=1048576 2023-12-03T11:30:42.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.013+0000 7fc553ded780 1 < canceled part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.1 size=1048576 2023-12-03T11:30:42.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.014+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~M1RHP3R5N9VZ.2 size=712122 2023-12-03T11:30:42.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.016+0000 7fc553ded780 1 < canceled multipart upload key=WOKG1DISIKK0 upload=M1RHP3R5N9VZ tag=1HIATFUFVSL9 type=write size=3857850 2023-12-03T11:30:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.017+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=D565M0U7ANPT type=del size=1697813 2023-12-03T11:30:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.018+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~PR3CF28IAO67.0 size=1048576 2023-12-03T11:30:42.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.018+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~PR3CF28IAO67.1 size=207077 2023-12-03T11:30:42.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.020+0000 7fc553ded780 1 < canceled multipart upload key=ICDF1EYNI5IU upload=PR3CF28IAO67 tag=8R23QKXAVL9Z type=write size=2304229 2023-12-03T11:30:42.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.022+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.0 size=1048576 2023-12-03T11:30:42.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.023+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.1 size=1048576 2023-12-03T11:30:42.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.025+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.2 size=70996 2023-12-03T11:30:42.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.025+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=WVWV7IDL0V3E tag=5R969U33JGQX type=write size=3216724 2023-12-03T11:30:42.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.026+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=70Q1PBDRWM92 type=del 2023-12-03T11:30:42.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.027+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=G6XPRHJBO44T type=write size=331887 2023-12-03T11:30:42.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.029+0000 7fc553ded780 1 < completed write operation key=FC4WU6SMNEWV tag=G5T8SMUV5OYM type=write size=295765 2023-12-03T11:30:42.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.030+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=VSS2OC9UU3KD type=write size=729165 2023-12-03T11:30:42.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.032+0000 7fc553ded780 1 < suggested update operation key=SWTRHXTAR0A1 tag=WQCGFXYHVXZZ type=del size=996330 2023-12-03T11:30:42.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.033+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=SCN91510PHQ9 type=del size=2872433 2023-12-03T11:30:42.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.035+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=YMKCCA0ZWUSX type=del 2023-12-03T11:30:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.037+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~IC29A552GZN0.0 size=238120 2023-12-03T11:30:42.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.037+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=IC29A552GZN0 tag=STHAYX0JO25K type=write size=1286696 2023-12-03T11:30:42.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.039+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=69334SUJF2DV type=write size=760776 2023-12-03T11:30:42.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.040+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=I33CVPM7MGAW type=write size=535524 2023-12-03T11:30:42.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.041+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=DMNFZ9JINVFM type=del 2023-12-03T11:30:42.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.042+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=VDZ2YLVUZKN9 type=del size=1140281 2023-12-03T11:30:42.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.045+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=7EJ00NWLKK4H type=write 2023-12-03T11:30:42.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.047+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~M00LR9W3CSGM.0 size=633772 2023-12-03T11:30:42.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.047+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=M00LR9W3CSGM tag=Z97TXBN373GL type=write size=1682348 2023-12-03T11:30:42.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.048+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.0 size=1048576 2023-12-03T11:30:42.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.050+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.1 size=1048576 2023-12-03T11:30:42.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.051+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~MTBQ3EETH2I8.2 size=797945 2023-12-03T11:30:42.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.053+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=MTBQ3EETH2I8 tag=NBF8N71WJ1MU type=write size=3943673 2023-12-03T11:30:42.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.054+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=F3SJIEVAVEPA type=del size=3599071 2023-12-03T11:30:42.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.056+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=0GGL3A6F9YSG type=write size=1014391 2023-12-03T11:30:42.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.058+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.0 size=1048576 2023-12-03T11:30:42.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.059+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.1 size=1048576 2023-12-03T11:30:42.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.060+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.2 size=1001421 2023-12-03T11:30:42.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.060+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=5FOKH98OEO9B tag=7T1UNV9531CV type=write size=4147149 2023-12-03T11:30:42.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.061+0000 7fc553ded780 1 < completed write operation key=588S2ASN9I0W tag=5P2ZYTIB51CR type=write size=538076 2023-12-03T11:30:42.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.062+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=YRRUS9E28BMO type=write size=940231 2023-12-03T11:30:42.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.066+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=ATI8RGZPKCJZ type=write 2023-12-03T11:30:42.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.067+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=WPRGO45HYZCF type=write size=439650 2023-12-03T11:30:42.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.068+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~SROTFP3G2REH.0 size=1048576 2023-12-03T11:30:42.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.068+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~SROTFP3G2REH.1 size=580404 2023-12-03T11:30:42.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.071+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=SROTFP3G2REH tag=2Z7GRUCUEJPH type=write size=2677556 2023-12-03T11:30:42.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.072+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=4DB9AUJRU602 type=del size=332421 2023-12-03T11:30:42.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.073+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=00SF054AJOGN type=del 2023-12-03T11:30:42.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.075+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=KKM3FM7TJJSC type=write size=77309 2023-12-03T11:30:42.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.076+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~J05J0RXGREIJ.0 size=1048576 2023-12-03T11:30:42.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.077+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~J05J0RXGREIJ.1 size=38127 2023-12-03T11:30:42.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.079+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=J05J0RXGREIJ tag=G7KJ3K97DCVU type=write size=2135279 2023-12-03T11:30:42.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.081+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~R50VMQ42MEDI.0 size=1048576 2023-12-03T11:30:42.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.082+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~R50VMQ42MEDI.1 size=464153 2023-12-03T11:30:42.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.082+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=R50VMQ42MEDI tag=PGRWQP9ABI6H type=write size=2561305 2023-12-03T11:30:42.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.084+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=D565M0U7ANPT type=del 2023-12-03T11:30:42.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.086+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.0 size=1048576 2023-12-03T11:30:42.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.087+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.1 size=1048576 2023-12-03T11:30:42.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.088+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.2 size=719960 2023-12-03T11:30:42.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.088+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=BZX82GVTECT0 tag=8OBRDY441VC3 type=write size=3865688 2023-12-03T11:30:42.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.089+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.0 size=1048576 2023-12-03T11:30:42.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.090+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.1 size=1048576 2023-12-03T11:30:42.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.091+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~WVWV7IDL0V3E.2 size=70996 2023-12-03T11:30:42.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.093+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=WVWV7IDL0V3E tag=5R969U33JGQX type=write size=3216724 2023-12-03T11:30:42.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.095+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.0 size=1048576 2023-12-03T11:30:42.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.096+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.1 size=1048576 2023-12-03T11:30:42.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.097+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.2 size=995682 2023-12-03T11:30:42.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.097+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=26OJVL7FEHRJ tag=DU3TZ5PPO5FR type=write size=4141410 2023-12-03T11:30:42.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.098+0000 7fc553ded780 1 < completed write operation key=FP2UZMDKYTD1 tag=G6XPRHJBO44T type=write size=331887 2023-12-03T11:30:42.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.100+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=EQTXWKYHVMYY type=del size=588783 2023-12-03T11:30:42.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.101+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=VSS2OC9UU3KD type=write size=729165 2023-12-03T11:30:42.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.102+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=O9TPGTUKRNST type=del size=3489142 2023-12-03T11:30:42.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.103+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=SCN91510PHQ9 type=del size=2872433 2023-12-03T11:30:42.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.104+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=OBDUAT1V1W5A type=write size=15568 2023-12-03T11:30:42.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.106+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~IC29A552GZN0.0 size=238120 2023-12-03T11:30:42.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.107+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=IC29A552GZN0 tag=STHAYX0JO25K type=write size=1286696 2023-12-03T11:30:42.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.109+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=6CXL36M6Z5ER type=write size=928842 2023-12-03T11:30:42.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.110+0000 7fc553ded780 -1 < failed to complete operation key=G1OBBD75M5ZQ tag=I33CVPM7MGAW type=write size=535524: (22) Invalid argument 2023-12-03T11:30:42.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.111+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=QGQZZX84UDOO type=del size=2587911 2023-12-03T11:30:42.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.112+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=VDZ2YLVUZKN9 type=del 2023-12-03T11:30:42.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.114+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~FG3U1X0K5I2B.0 size=1048576 2023-12-03T11:30:42.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.115+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~FG3U1X0K5I2B.1 size=399171 2023-12-03T11:30:42.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.115+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=FG3U1X0K5I2B tag=F3AYD5ZLNS75 type=write size=2496323 2023-12-03T11:30:42.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.116+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~M00LR9W3CSGM.0 size=633772 2023-12-03T11:30:42.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.118+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=M00LR9W3CSGM tag=Z97TXBN373GL type=write size=1682348 2023-12-03T11:30:42.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.120+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.0 size=1048576 2023-12-03T11:30:42.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.121+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.1 size=1048576 2023-12-03T11:30:42.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.122+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.2 size=652964 2023-12-03T11:30:42.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.122+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=A10VJ5BRZKBU tag=CV0ZP0WVYN7O type=write size=3798692 2023-12-03T11:30:42.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.123+0000 7fc553ded780 1 < canceled operation key=EK7V6U9DSLD0 tag=F3SJIEVAVEPA type=del size=3599071 2023-12-03T11:30:42.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.125+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=281LWAR1WWGS type=del size=3468204 2023-12-03T11:30:42.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.126+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.0 size=1048576 2023-12-03T11:30:42.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.127+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.1 size=1048576 2023-12-03T11:30:42.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.128+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FOKH98OEO9B.2 size=1001421 2023-12-03T11:30:42.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.130+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=5FOKH98OEO9B tag=7T1UNV9531CV type=write size=4147149 2023-12-03T11:30:42.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.132+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.0 size=1048576 2023-12-03T11:30:42.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.133+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.1 size=1048576 2023-12-03T11:30:42.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.134+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.2 size=570253 2023-12-03T11:30:42.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.134+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=CI4OQDYPRM0D tag=YI8SPSBVVYA6 type=write size=3715981 2023-12-03T11:30:42.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.137+0000 7fc553ded780 1 < suggested remove operation key=706CET5PSDEG tag=YRRUS9E28BMO type=write 2023-12-03T11:30:42.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.139+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~J73DI8RDY2JO.0 size=18124 2023-12-03T11:30:42.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.139+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=J73DI8RDY2JO tag=QT3SX3RMKA4Y type=write size=1066700 2023-12-03T11:30:42.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.141+0000 7fc553ded780 1 < canceled operation key=I3U08QX8MLBD tag=WPRGO45HYZCF type=write size=439650 2023-12-03T11:30:42.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.142+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=N7JKFFEL880Y type=write size=772540 2023-12-03T11:30:42.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.143+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=4DB9AUJRU602 type=del 2023-12-03T11:30:42.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.144+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=TVUDS305O9HO type=del size=2555485 2023-12-03T11:30:42.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.145+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=KKM3FM7TJJSC type=write size=77309 2023-12-03T11:30:42.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.146+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=Q9PZVHRNW7HK type=del size=4041663 2023-12-03T11:30:42.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.148+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~R50VMQ42MEDI.0 size=1048576 2023-12-03T11:30:42.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.149+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~R50VMQ42MEDI.1 size=464153 2023-12-03T11:30:42.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.151+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=R50VMQ42MEDI tag=PGRWQP9ABI6H type=write size=2561305 2023-12-03T11:30:42.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.152+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=XI0MYQIJEUVP type=del size=137840 2023-12-03T11:30:42.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.153+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.0 size=1048576 2023-12-03T11:30:42.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.155+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.1 size=1048576 2023-12-03T11:30:42.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.156+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~BZX82GVTECT0.2 size=719960 2023-12-03T11:30:42.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.158+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=BZX82GVTECT0 tag=8OBRDY441VC3 type=write size=3865688 2023-12-03T11:30:42.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.160+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~MG4P53BKFTQM.0 size=1048576 2023-12-03T11:30:42.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.161+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~MG4P53BKFTQM.1 size=309866 2023-12-03T11:30:42.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.161+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=MG4P53BKFTQM tag=EJ1MLTXTMK98 type=write size=2407018 2023-12-03T11:30:42.164 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.163+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.0 size=1048576 2023-12-03T11:30:42.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.164+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.1 size=1048576 2023-12-03T11:30:42.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.165+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~26OJVL7FEHRJ.2 size=995682 2023-12-03T11:30:42.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.167+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=26OJVL7FEHRJ tag=DU3TZ5PPO5FR type=write size=4141410 2023-12-03T11:30:42.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.168+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=NTD4QL0362N4 type=del size=429361 2023-12-03T11:30:42.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.172+0000 7fc553ded780 1 < suggested update operation key=SWTRHXTAR0A1 tag=EQTXWKYHVMYY type=del size=588783 2023-12-03T11:30:42.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.174+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=OAZ55OT12M2W type=del size=1957092 2023-12-03T11:30:42.176 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.175+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=O9TPGTUKRNST type=del 2023-12-03T11:30:42.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.178+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~2AR07PC1AHTI.0 size=1048576 2023-12-03T11:30:42.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.179+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~2AR07PC1AHTI.1 size=752994 2023-12-03T11:30:42.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.179+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=2AR07PC1AHTI tag=98VWDUPSFJ54 type=write size=2850146 2023-12-03T11:30:42.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.180+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=OBDUAT1V1W5A type=write size=15568 2023-12-03T11:30:42.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.182+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=5SJ5METQTR17 type=del size=2281978 2023-12-03T11:30:42.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.184+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=6CXL36M6Z5ER type=write 2023-12-03T11:30:42.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.187+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~EF9Z1667DIVR.0 size=589174 2023-12-03T11:30:42.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.187+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=EF9Z1667DIVR tag=L1G2FRS02H7Q type=write size=1637750 2023-12-03T11:30:42.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.188+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=QGQZZX84UDOO type=del 2023-12-03T11:30:42.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.190+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.0 size=1048576 2023-12-03T11:30:42.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.191+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.1 size=1048576 2023-12-03T11:30:42.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.192+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.2 size=629304 2023-12-03T11:30:42.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.192+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=3AY0M1KVUNB0 tag=ZVNVS7DB46F6 type=write size=3775032 2023-12-03T11:30:42.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.193+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~FG3U1X0K5I2B.0 size=1048576 2023-12-03T11:30:42.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.194+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~FG3U1X0K5I2B.1 size=399171 2023-12-03T11:30:42.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.197+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=FG3U1X0K5I2B tag=F3AYD5ZLNS75 type=write size=2496323 2023-12-03T11:30:42.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.198+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=P14UG4KM2PVF type=del size=520736 2023-12-03T11:30:42.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.199+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.0 size=1048576 2023-12-03T11:30:42.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.200+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.1 size=1048576 2023-12-03T11:30:42.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.201+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~A10VJ5BRZKBU.2 size=652964 2023-12-03T11:30:42.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.203+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=A10VJ5BRZKBU tag=CV0ZP0WVYN7O type=write size=3798692 2023-12-03T11:30:42.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.205+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~GKSUGTJPWW86.0 size=1048576 2023-12-03T11:30:42.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.206+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~GKSUGTJPWW86.1 size=339175 2023-12-03T11:30:42.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.206+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=GKSUGTJPWW86 tag=QLHXQVY4EDQS type=write size=2436327 2023-12-03T11:30:42.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.207+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=281LWAR1WWGS type=del 2023-12-03T11:30:42.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.208+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=DU9LBGECMYP8 type=del size=3162685 2023-12-03T11:30:42.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.210+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.0 size=1048576 2023-12-03T11:30:42.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.211+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.1 size=1048576 2023-12-03T11:30:42.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.212+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~CI4OQDYPRM0D.2 size=570253 2023-12-03T11:30:42.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.214+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=CI4OQDYPRM0D tag=YI8SPSBVVYA6 type=write size=3715981 2023-12-03T11:30:42.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.215+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=092CM3IRZG8I type=write size=966203 2023-12-03T11:30:42.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.217+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~J73DI8RDY2JO.0 size=18124 2023-12-03T11:30:42.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.219+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=J73DI8RDY2JO tag=QT3SX3RMKA4Y type=write size=1066700 2023-12-03T11:30:42.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.220+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=SDU8AEI3Y18P type=del size=415830 2023-12-03T11:30:42.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.221+0000 7fc553ded780 -1 < failed to complete operation key=CZP8OQRU641T tag=N7JKFFEL880Y type=write size=772540: (22) Invalid argument 2023-12-03T11:30:42.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.222+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=B2GBH5DDS189 type=del size=1996125 2023-12-03T11:30:42.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.223+0000 7fc553ded780 1 < canceled operation key=L4ME9RYRR4NG tag=TVUDS305O9HO type=del size=2555485 2023-12-03T11:30:42.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.225+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=Z9W310DMJCTL type=write size=155538 2023-12-03T11:30:42.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.226+0000 7fc553ded780 1 < canceled operation key=0H5OZMBMLQFJ tag=Q9PZVHRNW7HK type=del size=4041663 2023-12-03T11:30:42.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.227+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=JXCTT59ORWF3 type=del size=3771216 2023-12-03T11:30:42.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.230+0000 7fc553ded780 1 < suggested update operation key=WDV4CW863V33 tag=XI0MYQIJEUVP type=del size=137840 2023-12-03T11:30:42.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.231+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=MNMZSJSB9FJQ type=del size=172408 2023-12-03T11:30:42.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.232+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~MG4P53BKFTQM.0 size=1048576 2023-12-03T11:30:42.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.232+0000 7fc553ded780 1 < canceled part key=_multipart_WLTMDGRBF4C9.2~MG4P53BKFTQM.1 size=309866 2023-12-03T11:30:42.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.234+0000 7fc553ded780 1 < canceled multipart upload key=WLTMDGRBF4C9 upload=MG4P53BKFTQM tag=EJ1MLTXTMK98 type=write size=2407018 2023-12-03T11:30:42.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.235+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=1COYFK583QVW type=del size=3547695 2023-12-03T11:30:42.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.237+0000 7fc553ded780 1 < suggested remove operation key=JUYA4LXEQ4US tag=NTD4QL0362N4 type=del 2023-12-03T11:30:42.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.239+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=3P9L1G51W7MS type=del size=2296090 2023-12-03T11:30:42.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.240+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=OAZ55OT12M2W type=del 2023-12-03T11:30:42.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.241+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=D7ZVN80X6XJB type=del size=2093452 2023-12-03T11:30:42.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.242+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~2AR07PC1AHTI.0 size=1048576 2023-12-03T11:30:42.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.244+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~2AR07PC1AHTI.1 size=752994 2023-12-03T11:30:42.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.246+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=2AR07PC1AHTI tag=98VWDUPSFJ54 type=write size=2850146 2023-12-03T11:30:42.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.248+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~KY4TKLZVAHEO.0 size=1048576 2023-12-03T11:30:42.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.249+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~KY4TKLZVAHEO.1 size=965130 2023-12-03T11:30:42.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.249+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=KY4TKLZVAHEO tag=04XOC851K5ZV type=write size=3062282 2023-12-03T11:30:42.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.250+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=5SJ5METQTR17 type=del 2023-12-03T11:30:42.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.251+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=4A59R017EXKG type=del size=2771575 2023-12-03T11:30:42.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.253+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~EF9Z1667DIVR.0 size=589174 2023-12-03T11:30:42.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.256+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=EF9Z1667DIVR tag=L1G2FRS02H7Q type=write size=1637750 2023-12-03T11:30:42.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.257+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=GOZI1NGKKGK1 type=del size=573637 2023-12-03T11:30:42.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.258+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.0 size=1048576 2023-12-03T11:30:42.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.258+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.1 size=1048576 2023-12-03T11:30:42.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.260+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~3AY0M1KVUNB0.2 size=629304 2023-12-03T11:30:42.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.264+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=3AY0M1KVUNB0 tag=ZVNVS7DB46F6 type=write size=3775032 2023-12-03T11:30:42.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.265+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=LR6D4R1QTFB3 type=write size=528493 2023-12-03T11:30:42.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.267+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=P14UG4KM2PVF type=del 2023-12-03T11:30:42.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.269+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~ZKU37HZN7OMW.0 size=1048576 2023-12-03T11:30:42.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.270+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~ZKU37HZN7OMW.1 size=600800 2023-12-03T11:30:42.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.270+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=ZKU37HZN7OMW tag=G2JFG36U5URW type=write size=2697952 2023-12-03T11:30:42.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.271+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~GKSUGTJPWW86.0 size=1048576 2023-12-03T11:30:42.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.272+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~GKSUGTJPWW86.1 size=339175 2023-12-03T11:30:42.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.274+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=GKSUGTJPWW86 tag=QLHXQVY4EDQS type=write size=2436327 2023-12-03T11:30:42.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.275+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=H35Y3KP45UN4 type=del size=2605730 2023-12-03T11:30:42.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.277+0000 7fc553ded780 1 < canceled operation key=ZUNOFY5N3J3H tag=DU9LBGECMYP8 type=del size=3162685 2023-12-03T11:30:42.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.279+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~3WP3RJ1HXSDX.0 size=1048576 2023-12-03T11:30:42.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.280+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~3WP3RJ1HXSDX.1 size=1008719 2023-12-03T11:30:42.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.280+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=3WP3RJ1HXSDX tag=679K79657MMU type=write size=3105871 2023-12-03T11:30:42.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.283+0000 7fc553ded780 1 < suggested remove operation key=0H5OZMBMLQFJ tag=092CM3IRZG8I type=write 2023-12-03T11:30:42.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.285+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~Q9R8B1PMHL6N.0 size=1048576 2023-12-03T11:30:42.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.286+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~Q9R8B1PMHL6N.1 size=492447 2023-12-03T11:30:42.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.286+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=Q9R8B1PMHL6N tag=M333YX6MGJLN type=write size=2589599 2023-12-03T11:30:42.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.287+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=SDU8AEI3Y18P type=del 2023-12-03T11:30:42.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.288+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=FZ6AUKH1VUIU type=del size=3260173 2023-12-03T11:30:42.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.290+0000 7fc553ded780 -1 < failed to complete operation key=WDV4CW863V33 tag=B2GBH5DDS189 type=del size=1996125: (22) Invalid argument 2023-12-03T11:30:42.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.291+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=EC0YW15FB9DY type=del size=2212873 2023-12-03T11:30:42.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.292+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=Z9W310DMJCTL type=write size=155538 2023-12-03T11:30:42.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.295+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.0 size=1048576 2023-12-03T11:30:42.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.296+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.1 size=1048576 2023-12-03T11:30:42.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.296+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.2 size=851831 2023-12-03T11:30:42.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.296+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=M28L88LOGKHI tag=VXNLOEIEH39X type=write size=3997559 2023-12-03T11:30:42.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.298+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=JXCTT59ORWF3 type=del 2023-12-03T11:30:42.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.299+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=UHY4QERB84UE type=del size=674257 2023-12-03T11:30:42.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.300+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=MNMZSJSB9FJQ type=del 2023-12-03T11:30:42.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.303+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.0 size=1048576 2023-12-03T11:30:42.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.304+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.1 size=1048576 2023-12-03T11:30:42.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.305+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.2 size=508070 2023-12-03T11:30:42.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.305+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=75I05520CFSE tag=V198VRXHGHS1 type=write size=3653798 2023-12-03T11:30:42.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.306+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=1COYFK583QVW type=del 2023-12-03T11:30:42.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.308+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~H7S2V7AVUUWB.0 size=829710 2023-12-03T11:30:42.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.308+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=H7S2V7AVUUWB tag=KW93H6M64RKQ type=write size=1878286 2023-12-03T11:30:42.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.309+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=3P9L1G51W7MS type=del 2023-12-03T11:30:42.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.311+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=J8O63DXF876R type=del size=3861543 2023-12-03T11:30:42.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.312+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=D7ZVN80X6XJB type=del 2023-12-03T11:30:42.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.313+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=HC71TAKZIHC6 type=del size=1865868 2023-12-03T11:30:42.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.314+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~KY4TKLZVAHEO.0 size=1048576 2023-12-03T11:30:42.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.315+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~KY4TKLZVAHEO.1 size=965130 2023-12-03T11:30:42.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.317+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=KY4TKLZVAHEO tag=04XOC851K5ZV type=write size=3062282 2023-12-03T11:30:42.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.320+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.0 size=1048576 2023-12-03T11:30:42.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.321+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.1 size=1048576 2023-12-03T11:30:42.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.322+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.2 size=742671 2023-12-03T11:30:42.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.322+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=AW52Q8PY9ASC tag=G2S4I9TP08G2 type=write size=3888399 2023-12-03T11:30:42.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.323+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=4A59R017EXKG type=del 2023-12-03T11:30:42.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.325+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~GB4GVRNP1876.0 size=824926 2023-12-03T11:30:42.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.325+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=GB4GVRNP1876 tag=XRGALL4E26EB type=write size=1873502 2023-12-03T11:30:42.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.327+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=GOZI1NGKKGK1 type=del 2023-12-03T11:30:42.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.328+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=N73ZYJ7UJ9NK type=write size=294758 2023-12-03T11:30:42.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.329+0000 7fc553ded780 1 < completed write operation key=WDV4CW863V33 tag=LR6D4R1QTFB3 type=write size=528493 2023-12-03T11:30:42.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.330+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=HM7POWNGWSCA type=del size=2346926 2023-12-03T11:30:42.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.332+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~ZKU37HZN7OMW.0 size=1048576 2023-12-03T11:30:42.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.333+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~ZKU37HZN7OMW.1 size=600800 2023-12-03T11:30:42.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.335+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=ZKU37HZN7OMW tag=G2JFG36U5URW type=write size=2697952 2023-12-03T11:30:42.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.336+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=I8IQ0NH4YUPZ type=del size=1929311 2023-12-03T11:30:42.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.337+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=H35Y3KP45UN4 type=del 2023-12-03T11:30:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.338+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=SMT8NB4HSYCS type=del size=473338 2023-12-03T11:30:42.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.339+0000 7fc553ded780 1 < canceled part key=_multipart_CZP8OQRU641T.2~3WP3RJ1HXSDX.0 size=1048576 2023-12-03T11:30:42.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.340+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~3WP3RJ1HXSDX.1 size=1008719 2023-12-03T11:30:42.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.342+0000 7fc553ded780 1 < canceled multipart upload key=CZP8OQRU641T upload=3WP3RJ1HXSDX tag=679K79657MMU type=write size=3105871 2023-12-03T11:30:42.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.343+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=IAB9FQAHUETM type=del size=1803263 2023-12-03T11:30:42.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.344+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~Q9R8B1PMHL6N.0 size=1048576 2023-12-03T11:30:42.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.345+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~Q9R8B1PMHL6N.1 size=492447 2023-12-03T11:30:42.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.347+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=Q9R8B1PMHL6N tag=M333YX6MGJLN type=write size=2589599 2023-12-03T11:30:42.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.349+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~LHBJDRHD42ST.0 size=1048576 2023-12-03T11:30:42.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.350+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~LHBJDRHD42ST.1 size=428288 2023-12-03T11:30:42.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.350+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=LHBJDRHD42ST tag=BMQD1E6I699X type=write size=2525440 2023-12-03T11:30:42.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.351+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=FZ6AUKH1VUIU type=del 2023-12-03T11:30:42.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.352+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=I8OO8KCU8SCS type=del size=3557768 2023-12-03T11:30:42.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.353+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=EC0YW15FB9DY type=del 2023-12-03T11:30:42.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.355+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=YR1RZZUEK3HY type=del size=200387 2023-12-03T11:30:42.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.356+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.0 size=1048576 2023-12-03T11:30:42.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.357+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.1 size=1048576 2023-12-03T11:30:42.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.358+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~M28L88LOGKHI.2 size=851831 2023-12-03T11:30:42.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.360+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=M28L88LOGKHI tag=VXNLOEIEH39X type=write size=3997559 2023-12-03T11:30:42.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.361+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=6DFSBLPUK8WN type=del size=2733390 2023-12-03T11:30:42.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.362+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=UHY4QERB84UE type=del 2023-12-03T11:30:42.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.363+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=R9BI8CRW942W type=del size=2658874 2023-12-03T11:30:42.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.365+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.0 size=1048576 2023-12-03T11:30:42.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.365+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.1 size=1048576 2023-12-03T11:30:42.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.366+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~75I05520CFSE.2 size=508070 2023-12-03T11:30:42.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.369+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=75I05520CFSE tag=V198VRXHGHS1 type=write size=3653798 2023-12-03T11:30:42.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.371+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~6DH9ZK4UFBU2.0 size=1048576 2023-12-03T11:30:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.372+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~6DH9ZK4UFBU2.1 size=724005 2023-12-03T11:30:42.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.372+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=6DH9ZK4UFBU2 tag=S7JCWOEF7SGW type=write size=2821157 2023-12-03T11:30:42.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.373+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~H7S2V7AVUUWB.0 size=829710 2023-12-03T11:30:42.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.376+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=H7S2V7AVUUWB tag=KW93H6M64RKQ type=write size=1878286 2023-12-03T11:30:42.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.377+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=0EKAQFRCBHVU type=del size=3857326 2023-12-03T11:30:42.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.378+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=J8O63DXF876R type=del 2023-12-03T11:30:42.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.382+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~5OA0MNJBW0T5.0 size=1048576 2023-12-03T11:30:42.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.383+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~5OA0MNJBW0T5.1 size=255779 2023-12-03T11:30:42.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.383+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=5OA0MNJBW0T5 tag=0127YDQ52NC5 type=write size=2352931 2023-12-03T11:30:42.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.384+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=HC71TAKZIHC6 type=del size=1865868 2023-12-03T11:30:42.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.387+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~UUVQ0ZF83HOX.0 size=1048576 2023-12-03T11:30:42.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.388+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~UUVQ0ZF83HOX.1 size=351504 2023-12-03T11:30:42.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.388+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=UUVQ0ZF83HOX tag=DVW94U6UA2JN type=write size=2448656 2023-12-03T11:30:42.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.389+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.0 size=1048576 2023-12-03T11:30:42.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.390+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.1 size=1048576 2023-12-03T11:30:42.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.391+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~AW52Q8PY9ASC.2 size=742671 2023-12-03T11:30:42.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.394+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=AW52Q8PY9ASC tag=G2S4I9TP08G2 type=write size=3888399 2023-12-03T11:30:42.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.395+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=DWKVG4F86LR2 type=del size=3997862 2023-12-03T11:30:42.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.396+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~GB4GVRNP1876.0 size=824926 2023-12-03T11:30:42.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.398+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=GB4GVRNP1876 tag=XRGALL4E26EB type=write size=1873502 2023-12-03T11:30:42.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.401+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.0 size=1048576 2023-12-03T11:30:42.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.403+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.1 size=1048576 2023-12-03T11:30:42.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.404+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.2 size=320978 2023-12-03T11:30:42.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.404+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=YG4TIUEM9N18 tag=BSJP2N8OW2D7 type=write size=3466706 2023-12-03T11:30:42.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.405+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=N73ZYJ7UJ9NK type=write size=294758 2023-12-03T11:30:42.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.407+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=V4WQGX9MPG4F type=del size=382467 2023-12-03T11:30:42.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.408+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=HM7POWNGWSCA type=del 2023-12-03T11:30:42.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.410+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~TIMM2VSOY5CP.0 size=1048576 2023-12-03T11:30:42.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.411+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~TIMM2VSOY5CP.1 size=185193 2023-12-03T11:30:42.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.411+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=TIMM2VSOY5CP tag=25KA1HXSSH8U type=write size=2282345 2023-12-03T11:30:42.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.412+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=I8IQ0NH4YUPZ type=del 2023-12-03T11:30:42.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.414+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.0 size=1048576 2023-12-03T11:30:42.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.415+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.1 size=1048576 2023-12-03T11:30:42.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.416+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.2 size=719294 2023-12-03T11:30:42.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.416+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=BA2N7B8WX2U4 tag=RVF40BQ77AMG type=write size=3865022 2023-12-03T11:30:42.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.419+0000 7fc553ded780 1 < suggested remove operation key=RO0KGI2QMA71 tag=SMT8NB4HSYCS type=del 2023-12-03T11:30:42.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.421+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=G3MVUTWHO8ST type=del size=1772745 2023-12-03T11:30:42.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.422+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=IAB9FQAHUETM type=del 2023-12-03T11:30:42.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.423+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=U10H2F8GKW14 type=del size=531852 2023-12-03T11:30:42.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.424+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~LHBJDRHD42ST.0 size=1048576 2023-12-03T11:30:42.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.425+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~LHBJDRHD42ST.1 size=428288 2023-12-03T11:30:42.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.427+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=LHBJDRHD42ST tag=BMQD1E6I699X type=write size=2525440 2023-12-03T11:30:42.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.428+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=CA2N7OARNMDB type=del size=3774897 2023-12-03T11:30:42.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.429+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=I8OO8KCU8SCS type=del 2023-12-03T11:30:42.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.431+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=FU54O1W9O2K9 type=write size=26050 2023-12-03T11:30:42.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.433+0000 7fc553ded780 1 < suggested update operation key=WDV4CW863V33 tag=YR1RZZUEK3HY type=del size=200387 2023-12-03T11:30:42.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.434+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=CAT5FB2KS1MB type=del size=465853 2023-12-03T11:30:42.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.435+0000 7fc553ded780 1 < canceled operation key=C2TPF9Q5Z7XQ tag=6DFSBLPUK8WN type=del size=2733390 2023-12-03T11:30:42.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.438+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.0 size=1048576 2023-12-03T11:30:42.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.439+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.1 size=1048576 2023-12-03T11:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.440+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.2 size=475003 2023-12-03T11:30:42.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.440+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=YII8808P1BIL tag=GTCUN0YZWAN2 type=write size=3620731 2023-12-03T11:30:42.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.442+0000 7fc553ded780 1 < suggested remove operation key=FC4WU6SMNEWV tag=R9BI8CRW942W type=del 2023-12-03T11:30:42.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.443+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=ICHKL6PA1MTF type=del size=3479799 2023-12-03T11:30:42.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.444+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~6DH9ZK4UFBU2.0 size=1048576 2023-12-03T11:30:42.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.445+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~6DH9ZK4UFBU2.1 size=724005 2023-12-03T11:30:42.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.447+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=6DH9ZK4UFBU2 tag=S7JCWOEF7SGW type=write size=2821157 2023-12-03T11:30:42.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.449+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=CCAPL8WYZPEH type=write size=392452 2023-12-03T11:30:42.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.450+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=0EKAQFRCBHVU type=del 2023-12-03T11:30:42.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.452+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.0 size=1048576 2023-12-03T11:30:42.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.453+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.1 size=1048576 2023-12-03T11:30:42.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.454+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.2 size=263706 2023-12-03T11:30:42.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.454+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=PPCQCOFN13F9 tag=INZ8W7EW9BSZ type=write size=3409434 2023-12-03T11:30:42.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.455+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~5OA0MNJBW0T5.0 size=1048576 2023-12-03T11:30:42.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.456+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~5OA0MNJBW0T5.1 size=255779 2023-12-03T11:30:42.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.458+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=5OA0MNJBW0T5 tag=0127YDQ52NC5 type=write size=2352931 2023-12-03T11:30:42.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.460+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~5G1JXI3SDM0I.0 size=324679 2023-12-03T11:30:42.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.460+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=5G1JXI3SDM0I tag=BK3J19UZZVP7 type=write size=1373255 2023-12-03T11:30:42.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.461+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~UUVQ0ZF83HOX.0 size=1048576 2023-12-03T11:30:42.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.461+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~UUVQ0ZF83HOX.1 size=351504 2023-12-03T11:30:42.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.463+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=UUVQ0ZF83HOX tag=DVW94U6UA2JN type=write size=2448656 2023-12-03T11:30:42.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.465+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~O8PSYNSRRZZZ.0 size=589881 2023-12-03T11:30:42.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.465+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=O8PSYNSRRZZZ tag=4NIU5OX3ORSW type=write size=1638457 2023-12-03T11:30:42.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.467+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=DWKVG4F86LR2 type=del 2023-12-03T11:30:42.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.468+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=G3O4JVPQTQSL type=del size=201243 2023-12-03T11:30:42.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.469+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.0 size=1048576 2023-12-03T11:30:42.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.470+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.1 size=1048576 2023-12-03T11:30:42.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.472+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~YG4TIUEM9N18.2 size=320978 2023-12-03T11:30:42.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.474+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=YG4TIUEM9N18 tag=BSJP2N8OW2D7 type=write size=3466706 2023-12-03T11:30:42.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.476+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.0 size=1048576 2023-12-03T11:30:42.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.477+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.1 size=1048576 2023-12-03T11:30:42.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.478+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.2 size=347624 2023-12-03T11:30:42.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.478+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=70TG0L3B9TQX tag=KXPFEUN79HP8 type=write size=3493352 2023-12-03T11:30:42.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.480+0000 7fc553ded780 -1 < failed to complete operation key=RO0KGI2QMA71 tag=V4WQGX9MPG4F type=del size=382467: (22) Invalid argument 2023-12-03T11:30:42.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.482+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~YSODE0V9QYIO.0 size=130694 2023-12-03T11:30:42.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.482+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=YSODE0V9QYIO tag=WN82MHJZG42F type=write size=1179270 2023-12-03T11:30:42.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.483+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~TIMM2VSOY5CP.0 size=1048576 2023-12-03T11:30:42.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.483+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~TIMM2VSOY5CP.1 size=185193 2023-12-03T11:30:42.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.486+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=TIMM2VSOY5CP tag=25KA1HXSSH8U type=write size=2282345 2023-12-03T11:30:42.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.487+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=7XD76JS6BVX1 type=del size=4086925 2023-12-03T11:30:42.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.488+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.0 size=1048576 2023-12-03T11:30:42.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.489+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.1 size=1048576 2023-12-03T11:30:42.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.490+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~BA2N7B8WX2U4.2 size=719294 2023-12-03T11:30:42.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.493+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=BA2N7B8WX2U4 tag=RVF40BQ77AMG type=write size=3865022 2023-12-03T11:30:42.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.494+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=CAIW02IN7BZW type=write size=403184 2023-12-03T11:30:42.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.496+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=G3MVUTWHO8ST type=del 2023-12-03T11:30:42.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.499+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.0 size=1048576 2023-12-03T11:30:42.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.500+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.1 size=1048576 2023-12-03T11:30:42.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.501+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.2 size=866052 2023-12-03T11:30:42.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.501+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=VPSB8DVJU6R4 tag=DCQYGX63P2ZR type=write size=4011780 2023-12-03T11:30:42.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.504+0000 7fc553ded780 1 < suggested remove operation key=6OW429S51UWN tag=U10H2F8GKW14 type=del 2023-12-03T11:30:42.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.506+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.0 size=1048576 2023-12-03T11:30:42.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.507+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.1 size=1048576 2023-12-03T11:30:42.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.508+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.2 size=139687 2023-12-03T11:30:42.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.508+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=WC50HOJ82H4X tag=9D5Z7Y4ETO9W type=write size=3285415 2023-12-03T11:30:42.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.511+0000 7fc553ded780 1 < suggested remove operation key=UKAL8FCVIRSH tag=CA2N7OARNMDB type=del 2023-12-03T11:30:42.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.512+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=WZ9POOQTLILF type=del size=1223730 2023-12-03T11:30:42.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.514+0000 7fc553ded780 1 < canceled operation key=SWTRHXTAR0A1 tag=FU54O1W9O2K9 type=write size=26050 2023-12-03T11:30:42.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.516+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.0 size=1048576 2023-12-03T11:30:42.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.517+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.1 size=1048576 2023-12-03T11:30:42.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.518+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.2 size=911893 2023-12-03T11:30:42.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.518+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=73JEYEKG88AU tag=QRNLX649G2WY type=write size=4057621 2023-12-03T11:30:42.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.519+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=CAT5FB2KS1MB type=del 2023-12-03T11:30:42.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.521+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~CSQOE70E3CUA.0 size=1048576 2023-12-03T11:30:42.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.522+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~CSQOE70E3CUA.1 size=279472 2023-12-03T11:30:42.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.522+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=CSQOE70E3CUA tag=2APDO4NXI6VI type=write size=2376624 2023-12-03T11:30:42.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.523+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.0 size=1048576 2023-12-03T11:30:42.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.524+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.1 size=1048576 2023-12-03T11:30:42.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.526+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~YII8808P1BIL.2 size=475003 2023-12-03T11:30:42.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.528+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=YII8808P1BIL tag=GTCUN0YZWAN2 type=write size=3620731 2023-12-03T11:30:42.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.529+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=21P3M8VCGQU7 type=write size=326846 2023-12-03T11:30:42.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.531+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=ICHKL6PA1MTF type=del size=3479799 2023-12-03T11:30:42.533 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.533+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=LRHSN68KV1IK type=del size=2733413 2023-12-03T11:30:42.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.534+0000 7fc553ded780 1 < completed write operation key=PL22GPUFN3O0 tag=CCAPL8WYZPEH type=write size=392452 2023-12-03T11:30:42.536 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.535+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=QL3BY9JNPCO0 type=del size=1473548 2023-12-03T11:30:42.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.536+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.0 size=1048576 2023-12-03T11:30:42.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.537+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.1 size=1048576 2023-12-03T11:30:42.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.538+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PPCQCOFN13F9.2 size=263706 2023-12-03T11:30:42.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.540+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=PPCQCOFN13F9 tag=INZ8W7EW9BSZ type=write size=3409434 2023-12-03T11:30:42.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.542+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=S32H0R94MGK5 type=write size=954513 2023-12-03T11:30:42.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.543+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~5G1JXI3SDM0I.0 size=324679 2023-12-03T11:30:42.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.545+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=5G1JXI3SDM0I tag=BK3J19UZZVP7 type=write size=1373255 2023-12-03T11:30:42.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.546+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=Q18T3YJ67NT8 type=del size=2757483 2023-12-03T11:30:42.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.548+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~O8PSYNSRRZZZ.0 size=589881 2023-12-03T11:30:42.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.550+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=O8PSYNSRRZZZ tag=4NIU5OX3ORSW type=write size=1638457 2023-12-03T11:30:42.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.551+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=IU5Y8X7Y23E7 type=del size=3806636 2023-12-03T11:30:42.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.552+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=G3O4JVPQTQSL type=del 2023-12-03T11:30:42.554 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.553+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=A2JZ7AWUMDF5 type=del size=2299687 2023-12-03T11:30:42.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.554+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.0 size=1048576 2023-12-03T11:30:42.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.555+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.1 size=1048576 2023-12-03T11:30:42.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.555+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~70TG0L3B9TQX.2 size=347624 2023-12-03T11:30:42.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.557+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=70TG0L3B9TQX tag=KXPFEUN79HP8 type=write size=3493352 2023-12-03T11:30:42.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.559+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~VTS1G9815XKF.0 size=398115 2023-12-03T11:30:42.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.559+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=VTS1G9815XKF tag=D2EGRIJ9J50N type=write size=1446691 2023-12-03T11:30:42.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.560+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~YSODE0V9QYIO.0 size=130694 2023-12-03T11:30:42.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.562+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=YSODE0V9QYIO tag=WN82MHJZG42F type=write size=1179270 2023-12-03T11:30:42.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.564+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=8AGVI74IQZYV type=del size=4123112 2023-12-03T11:30:42.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.565+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=7XD76JS6BVX1 type=del 2023-12-03T11:30:42.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.566+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=01K4TMZ5WMIB type=write size=1041748 2023-12-03T11:30:42.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.569+0000 7fc553ded780 1 < suggested remove operation key=L4ME9RYRR4NG tag=CAIW02IN7BZW type=write 2023-12-03T11:30:42.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.570+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=5D0ANC021Q0Y type=write size=662863 2023-12-03T11:30:42.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.571+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.0 size=1048576 2023-12-03T11:30:42.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.572+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.1 size=1048576 2023-12-03T11:30:42.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.573+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~VPSB8DVJU6R4.2 size=866052 2023-12-03T11:30:42.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.575+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=VPSB8DVJU6R4 tag=DCQYGX63P2ZR type=write size=4011780 2023-12-03T11:30:42.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.577+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=0UGGH01V40MM type=del size=4114847 2023-12-03T11:30:42.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.578+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.0 size=1048576 2023-12-03T11:30:42.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.579+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.1 size=1048576 2023-12-03T11:30:42.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.580+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~WC50HOJ82H4X.2 size=139687 2023-12-03T11:30:42.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.582+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=WC50HOJ82H4X tag=9D5Z7Y4ETO9W type=write size=3285415 2023-12-03T11:30:42.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.584+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.0 size=1048576 2023-12-03T11:30:42.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.585+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.1 size=1048576 2023-12-03T11:30:42.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.586+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.2 size=159757 2023-12-03T11:30:42.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.586+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=IZN5S3VZN8OX tag=9H1B9TP42FDN type=write size=3305485 2023-12-03T11:30:42.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.587+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=WZ9POOQTLILF type=del 2023-12-03T11:30:42.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.588+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=3XR9IABQYPOZ type=del size=3884871 2023-12-03T11:30:42.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.589+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.0 size=1048576 2023-12-03T11:30:42.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.590+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.1 size=1048576 2023-12-03T11:30:42.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.591+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~73JEYEKG88AU.2 size=911893 2023-12-03T11:30:42.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.593+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=73JEYEKG88AU tag=QRNLX649G2WY type=write size=4057621 2023-12-03T11:30:42.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.595+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~DM62DMCN19RW.0 size=50484 2023-12-03T11:30:42.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.595+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=DM62DMCN19RW tag=9CX5Z04LJG99 type=write size=1099060 2023-12-03T11:30:42.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.596+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~CSQOE70E3CUA.0 size=1048576 2023-12-03T11:30:42.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.597+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~CSQOE70E3CUA.1 size=279472 2023-12-03T11:30:42.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.599+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=CSQOE70E3CUA tag=2APDO4NXI6VI type=write size=2376624 2023-12-03T11:30:42.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.601+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.0 size=1048576 2023-12-03T11:30:42.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.602+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.1 size=1048576 2023-12-03T11:30:42.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.603+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.2 size=910247 2023-12-03T11:30:42.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.603+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=XK2HG6QQD075 tag=KDYRGBZRPJTJ type=write size=4055975 2023-12-03T11:30:42.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.604+0000 7fc553ded780 1 < completed write operation key=ZHK2M78A0SLQ tag=21P3M8VCGQU7 type=write size=326846 2023-12-03T11:30:42.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.605+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=JXOSBJC7YK3K type=del size=3375350 2023-12-03T11:30:42.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.607+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=LRHSN68KV1IK type=del 2023-12-03T11:30:42.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.610+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~QVOA6VC563FU.0 size=141914 2023-12-03T11:30:42.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.610+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=QVOA6VC563FU tag=4HSGH8TVQ041 type=write size=1190490 2023-12-03T11:30:42.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.612+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=QL3BY9JNPCO0 type=del size=1473548 2023-12-03T11:30:42.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.613+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=QEINYOIIIIML type=del size=2742575 2023-12-03T11:30:42.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.614+0000 7fc553ded780 1 < completed write operation key=ICDF1EYNI5IU tag=S32H0R94MGK5 type=write size=954513 2023-12-03T11:30:42.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.616+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~1DAW8T21JT90.0 size=991826 2023-12-03T11:30:42.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.616+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=1DAW8T21JT90 tag=5QW7KQKLFHSZ type=write size=2040402 2023-12-03T11:30:42.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.618+0000 7fc553ded780 1 < suggested update operation key=EK7V6U9DSLD0 tag=Q18T3YJ67NT8 type=del size=2757483 2023-12-03T11:30:42.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.619+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=96EQ5FO7B4QY type=del size=2763866 2023-12-03T11:30:42.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.620+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=IU5Y8X7Y23E7 type=del 2023-12-03T11:30:42.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.622+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=SYEP0DSMUGSS type=del size=3744025 2023-12-03T11:30:42.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.623+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=A2JZ7AWUMDF5 type=del 2023-12-03T11:30:42.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.624+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=FMDONNC0W8YF type=del size=3762055 2023-12-03T11:30:42.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.625+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~VTS1G9815XKF.0 size=398115 2023-12-03T11:30:42.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.627+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=VTS1G9815XKF tag=D2EGRIJ9J50N type=write size=1446691 2023-12-03T11:30:42.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.628+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=8C1502TI38N1 type=del size=997705 2023-12-03T11:30:42.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.629+0000 7fc553ded780 1 < canceled operation key=6OW429S51UWN tag=8AGVI74IQZYV type=del size=4123112 2023-12-03T11:30:42.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.631+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~RRP2R9V7W2VX.0 size=1048576 2023-12-03T11:30:42.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.632+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~RRP2R9V7W2VX.1 size=414209 2023-12-03T11:30:42.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.632+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=RRP2R9V7W2VX tag=X93847WK85LU type=write size=2511361 2023-12-03T11:30:42.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.633+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=01K4TMZ5WMIB type=write size=1041748 2023-12-03T11:30:42.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.636+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~H4IIPEO6R9XJ.0 size=22104 2023-12-03T11:30:42.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.636+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=H4IIPEO6R9XJ tag=1YSNYI78Q4K8 type=write size=1070680 2023-12-03T11:30:42.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.637+0000 7fc553ded780 1 < completed write operation key=FC4WU6SMNEWV tag=5D0ANC021Q0Y type=write size=662863 2023-12-03T11:30:42.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.638+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=DWLMOTPTFFNW type=del size=362990 2023-12-03T11:30:42.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.639+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=0UGGH01V40MM type=del 2023-12-03T11:30:42.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.640+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=22S1L509QJ3E type=del size=3192164 2023-12-03T11:30:42.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.641+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.0 size=1048576 2023-12-03T11:30:42.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.642+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.1 size=1048576 2023-12-03T11:30:42.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.643+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~IZN5S3VZN8OX.2 size=159757 2023-12-03T11:30:42.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.645+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=IZN5S3VZN8OX tag=9H1B9TP42FDN type=write size=3305485 2023-12-03T11:30:42.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.647+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=88PZG1QLTWYD type=del size=3434317 2023-12-03T11:30:42.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.648+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=3XR9IABQYPOZ type=del 2023-12-03T11:30:42.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.649+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=YRZ8T537JOLC type=write size=848583 2023-12-03T11:30:42.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.650+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~DM62DMCN19RW.0 size=50484 2023-12-03T11:30:42.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.652+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=DM62DMCN19RW tag=9CX5Z04LJG99 type=write size=1099060 2023-12-03T11:30:42.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.654+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~V6J1TS3255DO.0 size=151561 2023-12-03T11:30:42.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.654+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=V6J1TS3255DO tag=KU29MWIBICP9 type=write size=1200137 2023-12-03T11:30:42.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.656+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.0 size=1048576 2023-12-03T11:30:42.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.657+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.1 size=1048576 2023-12-03T11:30:42.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.658+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~XK2HG6QQD075.2 size=910247 2023-12-03T11:30:42.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.660+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=XK2HG6QQD075 tag=KDYRGBZRPJTJ type=write size=4055975 2023-12-03T11:30:42.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.661+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=IHSTHHVNU52F type=del size=2133068 2023-12-03T11:30:42.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.662+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=JXOSBJC7YK3K type=del 2023-12-03T11:30:42.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.664+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.0 size=1048576 2023-12-03T11:30:42.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.665+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.1 size=1048576 2023-12-03T11:30:42.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.666+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.2 size=731776 2023-12-03T11:30:42.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.666+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=OSRCSPLHT2DC tag=H75BJUDEPTAF type=write size=3877504 2023-12-03T11:30:42.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.668+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~QVOA6VC563FU.0 size=141914 2023-12-03T11:30:42.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.670+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=QVOA6VC563FU tag=4HSGH8TVQ041 type=write size=1190490 2023-12-03T11:30:42.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.671+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=8PWQWK1XAAR7 type=write size=37220 2023-12-03T11:30:42.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.672+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=QEINYOIIIIML type=del 2023-12-03T11:30:42.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.673+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=O3QTBKYQ98KJ type=write size=198616 2023-12-03T11:30:42.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.675+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~1DAW8T21JT90.0 size=991826 2023-12-03T11:30:42.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.677+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=1DAW8T21JT90 tag=5QW7KQKLFHSZ type=write size=2040402 2023-12-03T11:30:42.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.678+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=19NZMP9365KF type=del size=894796 2023-12-03T11:30:42.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.679+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=96EQ5FO7B4QY type=del 2023-12-03T11:30:42.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.681+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~BFQ8S0QYV3FP.0 size=215009 2023-12-03T11:30:42.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.681+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=BFQ8S0QYV3FP tag=NE6H2BNDUTE1 type=write size=1263585 2023-12-03T11:30:42.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.682+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=SYEP0DSMUGSS type=del 2023-12-03T11:30:42.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.685+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.0 size=1048576 2023-12-03T11:30:42.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.686+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.1 size=1048576 2023-12-03T11:30:42.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.687+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.2 size=1042725 2023-12-03T11:30:42.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.687+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=CHVPHMFAHWOT tag=JMQUWIK4ZHKI type=write size=4188453 2023-12-03T11:30:42.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.688+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=FMDONNC0W8YF type=del 2023-12-03T11:30:42.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.689+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=5I5FHYN475JD type=write size=254831 2023-12-03T11:30:42.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.692+0000 7fc553ded780 1 < suggested update operation key=X741JIRT7RIG tag=8C1502TI38N1 type=del size=997705 2023-12-03T11:30:42.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.693+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=N66202I01C0P type=del size=4046642 2023-12-03T11:30:42.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.694+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~RRP2R9V7W2VX.0 size=1048576 2023-12-03T11:30:42.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.696+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~RRP2R9V7W2VX.1 size=414209 2023-12-03T11:30:42.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.698+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=RRP2R9V7W2VX tag=X93847WK85LU type=write size=2511361 2023-12-03T11:30:42.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.699+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=13IKMR3UD18I type=write size=234942 2023-12-03T11:30:42.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.701+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~H4IIPEO6R9XJ.0 size=22104 2023-12-03T11:30:42.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.703+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=H4IIPEO6R9XJ tag=1YSNYI78Q4K8 type=write size=1070680 2023-12-03T11:30:42.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.705+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~8EJU3U9OFU33.0 size=352132 2023-12-03T11:30:42.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.705+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=8EJU3U9OFU33 tag=6XV2GY8KY6KB type=write size=1400708 2023-12-03T11:30:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.707+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=DWLMOTPTFFNW type=del 2023-12-03T11:30:42.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.709+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.0 size=1048576 2023-12-03T11:30:42.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.710+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.1 size=1048576 2023-12-03T11:30:42.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.711+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.2 size=813357 2023-12-03T11:30:42.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.711+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=9K3BI5DHTD4G tag=VOVBLY26KBYW type=write size=3959085 2023-12-03T11:30:42.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.712+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=22S1L509QJ3E type=del 2023-12-03T11:30:42.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.715+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~GUE50QTRSTXM.0 size=480629 2023-12-03T11:30:42.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.715+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=GUE50QTRSTXM tag=63TOR0TS0Q8Z type=write size=1529205 2023-12-03T11:30:42.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.716+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=88PZG1QLTWYD type=del 2023-12-03T11:30:42.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.718+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=DNJABQNDG0IL type=del size=321496 2023-12-03T11:30:42.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.719+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=YRZ8T537JOLC type=write size=848583 2023-12-03T11:30:42.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.721+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O3JT30GTTZ7J.0 size=641531 2023-12-03T11:30:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.721+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=O3JT30GTTZ7J tag=4TG988J98OIZ type=write size=1690107 2023-12-03T11:30:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.723+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~V6J1TS3255DO.0 size=151561 2023-12-03T11:30:42.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.726+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=V6J1TS3255DO tag=KU29MWIBICP9 type=write size=1200137 2023-12-03T11:30:42.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.727+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=QXVT30RVW4TF type=del size=516066 2023-12-03T11:30:42.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.728+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=IHSTHHVNU52F type=del 2023-12-03T11:30:42.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.729+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=FDUJTCHQGJV7 type=del size=801384 2023-12-03T11:30:42.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.731+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.0 size=1048576 2023-12-03T11:30:42.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.732+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.1 size=1048576 2023-12-03T11:30:42.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.732+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~OSRCSPLHT2DC.2 size=731776 2023-12-03T11:30:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.734+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=OSRCSPLHT2DC tag=H75BJUDEPTAF type=write size=3877504 2023-12-03T11:30:42.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.736+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.0 size=1048576 2023-12-03T11:30:42.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.737+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.1 size=1048576 2023-12-03T11:30:42.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.738+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.2 size=907058 2023-12-03T11:30:42.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.738+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=R6TNR1L5DDS7 tag=Z7L1IPZ66ZJC type=write size=4052786 2023-12-03T11:30:42.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.741+0000 7fc553ded780 1 < suggested remove operation key=LRN5NYUZ5JT9 tag=8PWQWK1XAAR7 type=write 2023-12-03T11:30:42.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.743+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~76P2A3L97GDE.0 size=971565 2023-12-03T11:30:42.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.743+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=76P2A3L97GDE tag=ODKGP72I4ZC3 type=write size=2020141 2023-12-03T11:30:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.745+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=O3QTBKYQ98KJ type=write size=198616 2023-12-03T11:30:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.746+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=JCX2EEA8E0U7 type=del size=1193126 2023-12-03T11:30:42.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.749+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=19NZMP9365KF type=del 2023-12-03T11:30:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.751+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~DVOXISRP424N.0 size=1048576 2023-12-03T11:30:42.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.752+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~DVOXISRP424N.1 size=859316 2023-12-03T11:30:42.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.752+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=DVOXISRP424N tag=LAE0J9JPVVGO type=write size=2956468 2023-12-03T11:30:42.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.753+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~BFQ8S0QYV3FP.0 size=215009 2023-12-03T11:30:42.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.755+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=BFQ8S0QYV3FP tag=NE6H2BNDUTE1 type=write size=1263585 2023-12-03T11:30:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.756+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=08PS84GEVGO8 type=del size=2250321 2023-12-03T11:30:42.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.758+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.0 size=1048576 2023-12-03T11:30:42.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.759+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.1 size=1048576 2023-12-03T11:30:42.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.760+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CHVPHMFAHWOT.2 size=1042725 2023-12-03T11:30:42.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.762+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=CHVPHMFAHWOT tag=JMQUWIK4ZHKI type=write size=4188453 2023-12-03T11:30:42.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.763+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=5QURBVTN6D15 type=write size=1007202 2023-12-03T11:30:42.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.764+0000 7fc553ded780 1 < canceled operation key=YE73HBF64CS7 tag=5I5FHYN475JD type=write size=254831 2023-12-03T11:30:42.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.766+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~8UZ0RLEHV5ZD.0 size=479790 2023-12-03T11:30:42.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.766+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=8UZ0RLEHV5ZD tag=VWMY1RRVJF2B type=write size=1528366 2023-12-03T11:30:42.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.768+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=N66202I01C0P type=del 2023-12-03T11:30:42.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.769+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=4GC5M8ZF0L43 type=del size=3662026 2023-12-03T11:30:42.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.770+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=13IKMR3UD18I type=write size=234942 2023-12-03T11:30:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.771+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=CI93S84RIXW6 type=del size=826169 2023-12-03T11:30:42.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.771+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~8EJU3U9OFU33.0 size=352132 2023-12-03T11:30:42.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.773+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=8EJU3U9OFU33 tag=6XV2GY8KY6KB type=write size=1400708 2023-12-03T11:30:42.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.775+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.0 size=1048576 2023-12-03T11:30:42.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.776+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.1 size=1048576 2023-12-03T11:30:42.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.777+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.2 size=417981 2023-12-03T11:30:42.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.777+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=UCI99R0M2OQU tag=UN5U69R1CO1R type=write size=3563709 2023-12-03T11:30:42.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.778+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.0 size=1048576 2023-12-03T11:30:42.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.779+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.1 size=1048576 2023-12-03T11:30:42.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.780+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9K3BI5DHTD4G.2 size=813357 2023-12-03T11:30:42.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.782+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=9K3BI5DHTD4G tag=VOVBLY26KBYW type=write size=3959085 2023-12-03T11:30:42.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.784+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=LEQ5A4PH6PXK type=del size=359216 2023-12-03T11:30:42.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.785+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~GUE50QTRSTXM.0 size=480629 2023-12-03T11:30:42.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.787+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=GUE50QTRSTXM tag=63TOR0TS0Q8Z type=write size=1529205 2023-12-03T11:30:42.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.789+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~C62D869XQFKW.0 size=36574 2023-12-03T11:30:42.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.789+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=C62D869XQFKW tag=OAM3II6K7W77 type=write size=1085150 2023-12-03T11:30:42.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.790+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=DNJABQNDG0IL type=del 2023-12-03T11:30:42.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.791+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=Z02BHAGHSTNC type=del size=680036 2023-12-03T11:30:42.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.793+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~O3JT30GTTZ7J.0 size=641531 2023-12-03T11:30:42.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.795+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=O3JT30GTTZ7J tag=4TG988J98OIZ type=write size=1690107 2023-12-03T11:30:42.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.796+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=RDHEDZRQ68WZ type=del size=2000854 2023-12-03T11:30:42.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.797+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=QXVT30RVW4TF type=del 2023-12-03T11:30:42.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.798+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=VE1EGXA50SRA type=del size=2582655 2023-12-03T11:30:42.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.800+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=FDUJTCHQGJV7 type=del 2023-12-03T11:30:42.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.802+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~UPBPBX005J9F.0 size=1048576 2023-12-03T11:30:42.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.803+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~UPBPBX005J9F.1 size=63218 2023-12-03T11:30:42.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.803+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=UPBPBX005J9F tag=FF48U673MILU type=write size=2160370 2023-12-03T11:30:42.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.804+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.0 size=1048576 2023-12-03T11:30:42.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.805+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.1 size=1048576 2023-12-03T11:30:42.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.805+0000 7fc553ded780 1 < canceled part key=_multipart_J664UUE697Q8.2~R6TNR1L5DDS7.2 size=907058 2023-12-03T11:30:42.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.807+0000 7fc553ded780 1 < canceled multipart upload key=J664UUE697Q8 upload=R6TNR1L5DDS7 tag=Z7L1IPZ66ZJC type=write size=4052786 2023-12-03T11:30:42.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.810+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~C8YNENEYXVRT.0 size=1048576 2023-12-03T11:30:42.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.811+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~C8YNENEYXVRT.1 size=931891 2023-12-03T11:30:42.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.811+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=C8YNENEYXVRT tag=PMS20X2MAJKR type=write size=3029043 2023-12-03T11:30:42.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.812+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~76P2A3L97GDE.0 size=971565 2023-12-03T11:30:42.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.814+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=76P2A3L97GDE tag=ODKGP72I4ZC3 type=write size=2020141 2023-12-03T11:30:42.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.815+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=WM0NDF1I1BDS type=del size=2854574 2023-12-03T11:30:42.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.818+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=JCX2EEA8E0U7 type=del 2023-12-03T11:30:42.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.819+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=VTQY7H5YZFW9 type=del size=3284706 2023-12-03T11:30:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.820+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~DVOXISRP424N.0 size=1048576 2023-12-03T11:30:42.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.821+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~DVOXISRP424N.1 size=859316 2023-12-03T11:30:42.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.823+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=DVOXISRP424N tag=LAE0J9JPVVGO type=write size=2956468 2023-12-03T11:30:42.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.824+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=AN2CWUYM67N2 type=del size=2278633 2023-12-03T11:30:42.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.825+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=08PS84GEVGO8 type=del 2023-12-03T11:30:42.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.828+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~BKUMCI05VSU3.0 size=769549 2023-12-03T11:30:42.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.828+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=BKUMCI05VSU3 tag=86RTGT6YL4UR type=write size=1818125 2023-12-03T11:30:42.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.830+0000 7fc553ded780 1 < suggested update operation key=LRN5NYUZ5JT9 tag=5QURBVTN6D15 type=write size=1007202 2023-12-03T11:30:42.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.833+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.0 size=1048576 2023-12-03T11:30:42.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.834+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.1 size=1048576 2023-12-03T11:30:42.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.835+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.2 size=40414 2023-12-03T11:30:42.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.835+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=5SHF1FM2IZ3E tag=QEO2NTCLV7RS type=write size=3186142 2023-12-03T11:30:42.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.836+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~8UZ0RLEHV5ZD.0 size=479790 2023-12-03T11:30:42.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.838+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=8UZ0RLEHV5ZD tag=VWMY1RRVJF2B type=write size=1528366 2023-12-03T11:30:42.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.839+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=JU7EB908ALQB type=write size=821852 2023-12-03T11:30:42.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.842+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=4GC5M8ZF0L43 type=del 2023-12-03T11:30:42.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.844+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.0 size=1048576 2023-12-03T11:30:42.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.845+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.1 size=1048576 2023-12-03T11:30:42.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.846+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.2 size=468095 2023-12-03T11:30:42.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.846+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=11LXXBYX2IXE tag=SM55M5K3Z9QQ type=write size=3613823 2023-12-03T11:30:42.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.847+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=CI93S84RIXW6 type=del 2023-12-03T11:30:42.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.848+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=7IB60KLVI7LL type=del size=2483526 2023-12-03T11:30:42.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.850+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.0 size=1048576 2023-12-03T11:30:42.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.851+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.1 size=1048576 2023-12-03T11:30:42.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.852+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~UCI99R0M2OQU.2 size=417981 2023-12-03T11:30:42.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.854+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=UCI99R0M2OQU tag=UN5U69R1CO1R type=write size=3563709 2023-12-03T11:30:42.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.855+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=PF0E6JFVQD1F type=del size=2499088 2023-12-03T11:30:42.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.856+0000 7fc553ded780 -1 < failed to complete operation key=G1OBBD75M5ZQ tag=LEQ5A4PH6PXK type=del size=359216: (22) Invalid argument 2023-12-03T11:30:42.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.858+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=G3EWCL6J0MIM type=del size=3497186 2023-12-03T11:30:42.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.859+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~C62D869XQFKW.0 size=36574 2023-12-03T11:30:42.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.861+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=C62D869XQFKW tag=OAM3II6K7W77 type=write size=1085150 2023-12-03T11:30:42.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.862+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=Z33KTPCAIC4C type=write size=830020 2023-12-03T11:30:42.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.863+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=Z02BHAGHSTNC type=del 2023-12-03T11:30:42.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.866+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.0 size=1048576 2023-12-03T11:30:42.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.867+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.1 size=1048576 2023-12-03T11:30:42.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.868+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.2 size=104520 2023-12-03T11:30:42.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.868+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=RVRF4U4X7ACR tag=9S1P8SR5JNO1 type=write size=3250248 2023-12-03T11:30:42.870 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.869+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=RDHEDZRQ68WZ type=del 2023-12-03T11:30:42.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.870+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=EYDDPTS60C67 type=write size=598780 2023-12-03T11:30:42.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.872+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=VE1EGXA50SRA type=del 2023-12-03T11:30:42.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.873+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=WTSJ1HZW81UK type=write size=817733 2023-12-03T11:30:42.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.875+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~UPBPBX005J9F.0 size=1048576 2023-12-03T11:30:42.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.876+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~UPBPBX005J9F.1 size=63218 2023-12-03T11:30:42.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.878+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=UPBPBX005J9F tag=FF48U673MILU type=write size=2160370 2023-12-03T11:30:42.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.879+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=3O1G012V2SQN type=del size=974767 2023-12-03T11:30:42.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.880+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~C8YNENEYXVRT.0 size=1048576 2023-12-03T11:30:42.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.881+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~C8YNENEYXVRT.1 size=931891 2023-12-03T11:30:42.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.883+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=C8YNENEYXVRT tag=PMS20X2MAJKR type=write size=3029043 2023-12-03T11:30:42.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.884+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=3UO37TVP3QX4 type=del size=10849 2023-12-03T11:30:42.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.886+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=WM0NDF1I1BDS type=del 2023-12-03T11:30:42.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.887+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=12YB02T1TQD7 type=del size=2271592 2023-12-03T11:30:42.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.888+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=VTQY7H5YZFW9 type=del 2023-12-03T11:30:42.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.889+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=8Z0VSMY6HDF7 type=del size=3678665 2023-12-03T11:30:42.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.891+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=AN2CWUYM67N2 type=del 2023-12-03T11:30:42.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.893+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.0 size=1048576 2023-12-03T11:30:42.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.894+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.1 size=1048576 2023-12-03T11:30:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.895+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.2 size=29383 2023-12-03T11:30:42.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.895+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=4F66N53VOHGV tag=08PVCA2MVMZJ type=write size=3175111 2023-12-03T11:30:42.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.896+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~BKUMCI05VSU3.0 size=769549 2023-12-03T11:30:42.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.898+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=BKUMCI05VSU3 tag=86RTGT6YL4UR type=write size=1818125 2023-12-03T11:30:42.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.899+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=J3OPMR2R61XD type=write size=963949 2023-12-03T11:30:42.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.899+0000 7fc553ded780 1 < canceled part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.0 size=1048576 2023-12-03T11:30:42.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.900+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.1 size=1048576 2023-12-03T11:30:42.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.900+0000 7fc553ded780 1 < canceled part key=_multipart_0H5OZMBMLQFJ.2~5SHF1FM2IZ3E.2 size=40414 2023-12-03T11:30:42.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.902+0000 7fc553ded780 1 < canceled multipart upload key=0H5OZMBMLQFJ upload=5SHF1FM2IZ3E tag=QEO2NTCLV7RS type=write size=3186142 2023-12-03T11:30:42.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.904+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=W5OPZUP74V6R type=del size=2951943 2023-12-03T11:30:42.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.905+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=JU7EB908ALQB type=write size=821852 2023-12-03T11:30:42.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.906+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=T3HOLJWZ9BLO type=del size=2466274 2023-12-03T11:30:42.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.907+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.0 size=1048576 2023-12-03T11:30:42.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.908+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.1 size=1048576 2023-12-03T11:30:42.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.910+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~11LXXBYX2IXE.2 size=468095 2023-12-03T11:30:42.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.912+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=11LXXBYX2IXE tag=SM55M5K3Z9QQ type=write size=3613823 2023-12-03T11:30:42.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.913+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=TLAGNGWK6K0T type=del size=631994 2023-12-03T11:30:42.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.915+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=7IB60KLVI7LL type=del 2023-12-03T11:30:42.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.917+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.0 size=1048576 2023-12-03T11:30:42.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.918+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.1 size=1048576 2023-12-03T11:30:42.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.919+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.2 size=560251 2023-12-03T11:30:42.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.919+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=Z22D84C4RQRB tag=MLF8FDJOFG89 type=write size=3705979 2023-12-03T11:30:42.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.922+0000 7fc553ded780 1 < suggested remove operation key=L4ME9RYRR4NG tag=PF0E6JFVQD1F type=del 2023-12-03T11:30:42.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.923+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=G6TLBGOHRMQ5 type=del size=1587474 2023-12-03T11:30:42.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.924+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=G3EWCL6J0MIM type=del 2023-12-03T11:30:42.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.925+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=C1MD6D7HYSSG type=del size=1822477 2023-12-03T11:30:42.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.927+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=Z33KTPCAIC4C type=write size=830020 2023-12-03T11:30:42.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.929+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~MZKHO0H7CSKV.0 size=1048576 2023-12-03T11:30:42.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.930+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~MZKHO0H7CSKV.1 size=69780 2023-12-03T11:30:42.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.930+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=MZKHO0H7CSKV tag=4A0BBMXG5JJO type=write size=2166932 2023-12-03T11:30:42.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.931+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.0 size=1048576 2023-12-03T11:30:42.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.932+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.1 size=1048576 2023-12-03T11:30:42.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.933+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~RVRF4U4X7ACR.2 size=104520 2023-12-03T11:30:42.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.935+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=RVRF4U4X7ACR tag=9S1P8SR5JNO1 type=write size=3250248 2023-12-03T11:30:42.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.938+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.0 size=1048576 2023-12-03T11:30:42.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.941+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.1 size=1048576 2023-12-03T11:30:42.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.941+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.2 size=44654 2023-12-03T11:30:42.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.941+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=CN7TZWGHO2IU tag=16RWKFD27XQ2 type=write size=3190382 2023-12-03T11:30:42.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.944+0000 7fc553ded780 1 < suggested remove operation key=WLTMDGRBF4C9 tag=EYDDPTS60C67 type=write 2023-12-03T11:30:42.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.945+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=NPHWPGVWR172 type=del size=2401262 2023-12-03T11:30:42.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.946+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=WTSJ1HZW81UK type=write size=817733 2023-12-03T11:30:42.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.947+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=CFGB769C19N1 type=write size=653434 2023-12-03T11:30:42.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.950+0000 7fc553ded780 1 < suggested remove operation key=FP2UZMDKYTD1 tag=3O1G012V2SQN type=del 2023-12-03T11:30:42.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.951+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=GT77VW0HCV5Q type=write size=431534 2023-12-03T11:30:42.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.952+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=3UO37TVP3QX4 type=del 2023-12-03T11:30:42.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.954+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.0 size=1048576 2023-12-03T11:30:42.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.955+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.1 size=1048576 2023-12-03T11:30:42.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.956+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.2 size=902721 2023-12-03T11:30:42.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.956+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=R5873NG7F060 tag=46092FUSYGOS type=write size=4048449 2023-12-03T11:30:42.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.959+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=12YB02T1TQD7 type=del 2023-12-03T11:30:42.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.961+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=B94QKPNLV81S type=del size=2865743 2023-12-03T11:30:42.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.963+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=8Z0VSMY6HDF7 type=del 2023-12-03T11:30:42.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.965+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=9I8FN2P75S25 type=del size=4088131 2023-12-03T11:30:42.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.968+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.0 size=1048576 2023-12-03T11:30:42.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.968+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.1 size=1048576 2023-12-03T11:30:42.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.971+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~4F66N53VOHGV.2 size=29383 2023-12-03T11:30:42.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.974+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=4F66N53VOHGV tag=08PVCA2MVMZJ type=write size=3175111 2023-12-03T11:30:42.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.976+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~3280WABS2BXH.0 size=276601 2023-12-03T11:30:42.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.976+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=3280WABS2BXH tag=CISYO5M86U6F type=write size=1325177 2023-12-03T11:30:42.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.978+0000 7fc553ded780 1 < completed write operation key=TQRZT37AB9WC tag=J3OPMR2R61XD type=write size=963949 2023-12-03T11:30:42.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.980+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.0 size=1048576 2023-12-03T11:30:42.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.981+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.1 size=1048576 2023-12-03T11:30:42.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.982+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.2 size=39273 2023-12-03T11:30:42.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.982+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=2G3EYKZTLLE5 tag=H4FWQPIF4R24 type=write size=3185001 2023-12-03T11:30:42.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.983+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=W5OPZUP74V6R type=del 2023-12-03T11:30:42.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.984+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=GTRRDQARSWMD type=del size=644852 2023-12-03T11:30:42.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.986+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=T3HOLJWZ9BLO type=del 2023-12-03T11:30:42.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.987+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=8KF1ZH2D6AZJ type=del size=607175 2023-12-03T11:30:42.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.990+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=TLAGNGWK6K0T type=del 2023-12-03T11:30:42.993 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.992+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.0 size=1048576 2023-12-03T11:30:42.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.993+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.1 size=1048576 2023-12-03T11:30:42.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.994+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.2 size=695623 2023-12-03T11:30:42.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.994+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=4F59WF0P48EV tag=XAHICSMT2M3A type=write size=3841351 2023-12-03T11:30:42.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.995+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.0 size=1048576 2023-12-03T11:30:42.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.996+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.1 size=1048576 2023-12-03T11:30:42.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.997+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~Z22D84C4RQRB.2 size=560251 2023-12-03T11:30:43.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:42.999+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=Z22D84C4RQRB tag=MLF8FDJOFG89 type=write size=3705979 2023-12-03T11:30:43.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.000+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=SV98AFC2JKGR type=write size=767342 2023-12-03T11:30:43.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.002+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=G6TLBGOHRMQ5 type=del 2023-12-03T11:30:43.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.004+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~G3FRMDBTKAAZ.0 size=530599 2023-12-03T11:30:43.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.004+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=G3FRMDBTKAAZ tag=33DZB04JS31U type=write size=1579175 2023-12-03T11:30:43.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.005+0000 7fc553ded780 -1 < failed to complete operation key=WLTMDGRBF4C9 tag=C1MD6D7HYSSG type=del size=1822477: (22) Invalid argument 2023-12-03T11:30:43.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.007+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~ZT7TY3R39CAQ.0 size=748383 2023-12-03T11:30:43.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.007+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=ZT7TY3R39CAQ tag=DQ0GP9UP0V1M type=write size=1796959 2023-12-03T11:30:43.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.009+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~MZKHO0H7CSKV.0 size=1048576 2023-12-03T11:30:43.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.010+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~MZKHO0H7CSKV.1 size=69780 2023-12-03T11:30:43.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.012+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=MZKHO0H7CSKV tag=4A0BBMXG5JJO type=write size=2166932 2023-12-03T11:30:43.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.013+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=9Q7EXKED4MBD type=del size=3759835 2023-12-03T11:30:43.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.014+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.0 size=1048576 2023-12-03T11:30:43.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.015+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.1 size=1048576 2023-12-03T11:30:43.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.015+0000 7fc553ded780 1 < canceled part key=_multipart_WLTMDGRBF4C9.2~CN7TZWGHO2IU.2 size=44654 2023-12-03T11:30:43.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.017+0000 7fc553ded780 1 < canceled multipart upload key=WLTMDGRBF4C9 upload=CN7TZWGHO2IU tag=16RWKFD27XQ2 type=write size=3190382 2023-12-03T11:30:43.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.020+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~W9FRJCRS50MV.0 size=1048576 2023-12-03T11:30:43.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.021+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~W9FRJCRS50MV.1 size=511465 2023-12-03T11:30:43.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.021+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=W9FRJCRS50MV tag=LASDPTODGH2J type=write size=2608617 2023-12-03T11:30:43.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.022+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=NPHWPGVWR172 type=del 2023-12-03T11:30:43.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.023+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=ZZFXJ1CVQ4SN type=write size=30685 2023-12-03T11:30:43.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.024+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=CFGB769C19N1 type=write size=653434 2023-12-03T11:30:43.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.026+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=0CQOEHX21PSZ type=del size=2685068 2023-12-03T11:30:43.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.027+0000 7fc553ded780 1 < completed write operation key=ICDF1EYNI5IU tag=GT77VW0HCV5Q type=write size=431534 2023-12-03T11:30:43.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.029+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~GFLY49BFS7OZ.0 size=572003 2023-12-03T11:30:43.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.029+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=GFLY49BFS7OZ tag=G4CSK5DR17GW type=write size=1620579 2023-12-03T11:30:43.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.031+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.0 size=1048576 2023-12-03T11:30:43.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.032+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.1 size=1048576 2023-12-03T11:30:43.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.033+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~R5873NG7F060.2 size=902721 2023-12-03T11:30:43.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.035+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=R5873NG7F060 tag=46092FUSYGOS type=write size=4048449 2023-12-03T11:30:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.036+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=1M9TPWSV1MOE type=write size=769686 2023-12-03T11:30:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.039+0000 7fc553ded780 1 < suggested update operation key=EK7V6U9DSLD0 tag=B94QKPNLV81S type=del size=2865743 2023-12-03T11:30:43.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.040+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=0FP2RT2FNNJB type=write size=922629 2023-12-03T11:30:43.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.041+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=9I8FN2P75S25 type=del 2023-12-03T11:30:43.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.043+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~YJB8QSVQWWQR.0 size=426120 2023-12-03T11:30:43.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.043+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=YJB8QSVQWWQR tag=L3RJVS1ANGRL type=write size=1474696 2023-12-03T11:30:43.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.044+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~3280WABS2BXH.0 size=276601 2023-12-03T11:30:43.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.046+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=3280WABS2BXH tag=CISYO5M86U6F type=write size=1325177 2023-12-03T11:30:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.048+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=CW8NE5U04FKJ type=del size=194168 2023-12-03T11:30:43.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.049+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.0 size=1048576 2023-12-03T11:30:43.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.050+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.1 size=1048576 2023-12-03T11:30:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.051+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~2G3EYKZTLLE5.2 size=39273 2023-12-03T11:30:43.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.053+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=2G3EYKZTLLE5 tag=H4FWQPIF4R24 type=write size=3185001 2023-12-03T11:30:43.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.054+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=8C66BE51QAIF type=del size=2328461 2023-12-03T11:30:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.058+0000 7fc553ded780 1 < suggested update operation key=WDV4CW863V33 tag=GTRRDQARSWMD type=del size=644852 2023-12-03T11:30:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.060+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.0 size=1048576 2023-12-03T11:30:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.061+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.1 size=1048576 2023-12-03T11:30:43.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.062+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.2 size=563051 2023-12-03T11:30:43.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.062+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=4WGFG70IOEK4 tag=ULD1P0186BIX type=write size=3708779 2023-12-03T11:30:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.064+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=8KF1ZH2D6AZJ type=del 2023-12-03T11:30:43.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.065+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=7W89YU6TKX64 type=del size=1235631 2023-12-03T11:30:43.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.066+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.0 size=1048576 2023-12-03T11:30:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.068+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.1 size=1048576 2023-12-03T11:30:43.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.069+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~4F59WF0P48EV.2 size=695623 2023-12-03T11:30:43.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.071+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=4F59WF0P48EV tag=XAHICSMT2M3A type=write size=3841351 2023-12-03T11:30:43.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.073+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~SBE4GMU6P6W1.0 size=1048576 2023-12-03T11:30:43.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.074+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~SBE4GMU6P6W1.1 size=206357 2023-12-03T11:30:43.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.074+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=SBE4GMU6P6W1 tag=TYEQ0B3P1FGT type=write size=2303509 2023-12-03T11:30:43.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.075+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=SV98AFC2JKGR type=write size=767342 2023-12-03T11:30:43.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.077+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=AANZ7WHNZ4FY type=write size=720907 2023-12-03T11:30:43.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.078+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~G3FRMDBTKAAZ.0 size=530599 2023-12-03T11:30:43.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.080+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=G3FRMDBTKAAZ tag=33DZB04JS31U type=write size=1579175 2023-12-03T11:30:43.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.081+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=4ADUQBFEUFWF type=del size=2875687 2023-12-03T11:30:43.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.083+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~ZT7TY3R39CAQ.0 size=748383 2023-12-03T11:30:43.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.086+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=ZT7TY3R39CAQ tag=DQ0GP9UP0V1M type=write size=1796959 2023-12-03T11:30:43.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.087+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=M48RLXAX07YP type=del size=2881432 2023-12-03T11:30:43.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.088+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=9Q7EXKED4MBD type=del 2023-12-03T11:30:43.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.090+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~0RZD8JG6W2KL.0 size=1048576 2023-12-03T11:30:43.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.091+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~0RZD8JG6W2KL.1 size=485826 2023-12-03T11:30:43.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.091+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=0RZD8JG6W2KL tag=44IUPHPFIQBO type=write size=2582978 2023-12-03T11:30:43.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.091+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~W9FRJCRS50MV.0 size=1048576 2023-12-03T11:30:43.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.092+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~W9FRJCRS50MV.1 size=511465 2023-12-03T11:30:43.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.094+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=W9FRJCRS50MV tag=LASDPTODGH2J type=write size=2608617 2023-12-03T11:30:43.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.097+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.0 size=1048576 2023-12-03T11:30:43.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.097+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.1 size=1048576 2023-12-03T11:30:43.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.098+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.2 size=911060 2023-12-03T11:30:43.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.098+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=Z0N2ONDNOZEW tag=3IOO1NR52FRS type=write size=4056788 2023-12-03T11:30:43.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.101+0000 7fc553ded780 1 < suggested remove operation key=FC4WU6SMNEWV tag=ZZFXJ1CVQ4SN type=write 2023-12-03T11:30:43.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.102+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=X5FQ4631S9QD type=write size=131800 2023-12-03T11:30:43.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.104+0000 7fc553ded780 1 < suggested update operation key=L4ME9RYRR4NG tag=0CQOEHX21PSZ type=del size=2685068 2023-12-03T11:30:43.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.105+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=FYXRZM8QIHY1 type=del size=377905 2023-12-03T11:30:43.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.107+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~GFLY49BFS7OZ.0 size=572003 2023-12-03T11:30:43.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.109+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=GFLY49BFS7OZ tag=G4CSK5DR17GW type=write size=1620579 2023-12-03T11:30:43.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.110+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=WU8YWJIMRX5G type=del size=549388 2023-12-03T11:30:43.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.112+0000 7fc553ded780 1 < suggested update operation key=IRMADF9MXXBZ tag=1M9TPWSV1MOE type=write size=769686 2023-12-03T11:30:43.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.113+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=4GM6E97M2XKO type=del size=3077017 2023-12-03T11:30:43.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.114+0000 7fc553ded780 1 < completed write operation key=X741JIRT7RIG tag=0FP2RT2FNNJB type=write size=922629 2023-12-03T11:30:43.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.116+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~C3RPQGBD81L1.0 size=1048576 2023-12-03T11:30:43.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.117+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~C3RPQGBD81L1.1 size=307187 2023-12-03T11:30:43.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.117+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=C3RPQGBD81L1 tag=8DFRBB90BOQD type=write size=2404339 2023-12-03T11:30:43.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.119+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~YJB8QSVQWWQR.0 size=426120 2023-12-03T11:30:43.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.121+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=YJB8QSVQWWQR tag=L3RJVS1ANGRL type=write size=1474696 2023-12-03T11:30:43.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.123+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.0 size=1048576 2023-12-03T11:30:43.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.124+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.1 size=1048576 2023-12-03T11:30:43.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.125+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.2 size=227182 2023-12-03T11:30:43.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.125+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=KZROCS408SS2 tag=18AQVNEY8UEB type=write size=3372910 2023-12-03T11:30:43.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.127+0000 7fc553ded780 1 < suggested update operation key=ZHK2M78A0SLQ tag=CW8NE5U04FKJ type=del size=194168 2023-12-03T11:30:43.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.128+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=WF08UKNI4G2G type=del size=1157881 2023-12-03T11:30:43.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.129+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=8C66BE51QAIF type=del 2023-12-03T11:30:43.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.131+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.0 size=1048576 2023-12-03T11:30:43.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.132+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.1 size=1048576 2023-12-03T11:30:43.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.133+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.2 size=873324 2023-12-03T11:30:43.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.133+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=AUQMPWBE3JZL tag=YHHPI9P8E5HC type=write size=4019052 2023-12-03T11:30:43.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.134+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.0 size=1048576 2023-12-03T11:30:43.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.135+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.1 size=1048576 2023-12-03T11:30:43.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.135+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~4WGFG70IOEK4.2 size=563051 2023-12-03T11:30:43.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.137+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=4WGFG70IOEK4 tag=ULD1P0186BIX type=write size=3708779 2023-12-03T11:30:43.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.139+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~9JW1IXCE9H46.0 size=1048576 2023-12-03T11:30:43.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.140+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~9JW1IXCE9H46.1 size=45150 2023-12-03T11:30:43.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.140+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=9JW1IXCE9H46 tag=BJDPBO2LW3LZ type=write size=2142302 2023-12-03T11:30:43.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.142+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=7W89YU6TKX64 type=del 2023-12-03T11:30:43.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.143+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=6IM8J2FK6CAL type=del size=152960 2023-12-03T11:30:43.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.144+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~SBE4GMU6P6W1.0 size=1048576 2023-12-03T11:30:43.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.145+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~SBE4GMU6P6W1.1 size=206357 2023-12-03T11:30:43.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.147+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=SBE4GMU6P6W1 tag=TYEQ0B3P1FGT type=write size=2303509 2023-12-03T11:30:43.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.149+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.0 size=1048576 2023-12-03T11:30:43.151 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.150+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.1 size=1048576 2023-12-03T11:30:43.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.151+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.2 size=735710 2023-12-03T11:30:43.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.151+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=HJRSV90BSXHS tag=OKOYXEC7QSNQ type=write size=3881438 2023-12-03T11:30:43.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.153+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=AANZ7WHNZ4FY type=write size=720907 2023-12-03T11:30:43.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.154+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=LN7W3TT0A8J5 type=write size=320564 2023-12-03T11:30:43.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.155+0000 7fc553ded780 1 < canceled operation key=C2TPF9Q5Z7XQ tag=4ADUQBFEUFWF type=del size=2875687 2023-12-03T11:30:43.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.156+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=Q27H4IP5A7DN type=del size=2867228 2023-12-03T11:30:43.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.158+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=M48RLXAX07YP type=del 2023-12-03T11:30:43.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.161+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~1FTIHPQO8E2D.0 size=762636 2023-12-03T11:30:43.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.161+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=1FTIHPQO8E2D tag=30VR6PJ8HXS2 type=write size=1811212 2023-12-03T11:30:43.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.162+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~0RZD8JG6W2KL.0 size=1048576 2023-12-03T11:30:43.164 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.163+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~0RZD8JG6W2KL.1 size=485826 2023-12-03T11:30:43.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.165+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=0RZD8JG6W2KL tag=44IUPHPFIQBO type=write size=2582978 2023-12-03T11:30:43.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.167+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.0 size=1048576 2023-12-03T11:30:43.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.168+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.1 size=1048576 2023-12-03T11:30:43.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.169+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.2 size=793229 2023-12-03T11:30:43.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.169+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=5U3J4RR00AGP tag=1YNHQ7VKWK17 type=write size=3938957 2023-12-03T11:30:43.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.170+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.0 size=1048576 2023-12-03T11:30:43.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.170+0000 7fc553ded780 1 < canceled part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.1 size=1048576 2023-12-03T11:30:43.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.171+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~Z0N2ONDNOZEW.2 size=911060 2023-12-03T11:30:43.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.173+0000 7fc553ded780 1 < canceled multipart upload key=ZHK2M78A0SLQ upload=Z0N2ONDNOZEW tag=3IOO1NR52FRS type=write size=4056788 2023-12-03T11:30:43.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.176+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~C3KI7G1NGYJF.0 size=918692 2023-12-03T11:30:43.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.176+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=C3KI7G1NGYJF tag=EO0HRD7B5Y6X type=write size=1967268 2023-12-03T11:30:43.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.177+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=X5FQ4631S9QD type=write size=131800 2023-12-03T11:30:43.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.179+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~A0JE3U18Z28E.0 size=1048576 2023-12-03T11:30:43.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.180+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~A0JE3U18Z28E.1 size=476054 2023-12-03T11:30:43.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.180+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=A0JE3U18Z28E tag=JWZZUYBR95AD type=write size=2573206 2023-12-03T11:30:43.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.182+0000 7fc553ded780 -1 < failed to complete operation key=ZHK2M78A0SLQ tag=FYXRZM8QIHY1 type=del size=377905: (22) Invalid argument 2023-12-03T11:30:43.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.184+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.0 size=1048576 2023-12-03T11:30:43.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.185+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.1 size=1048576 2023-12-03T11:30:43.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.186+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.2 size=1046405 2023-12-03T11:30:43.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.186+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=VKK5MJVBWAFP tag=GIHUMMNDZX2V type=write size=4192133 2023-12-03T11:30:43.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.187+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=WU8YWJIMRX5G type=del 2023-12-03T11:30:43.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.189+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=PLIOA8EX81PV type=del size=1714468 2023-12-03T11:30:43.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.190+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=4GM6E97M2XKO type=del 2023-12-03T11:30:43.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.192+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=AN0PVHHIRHHO type=del size=3048593 2023-12-03T11:30:43.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.193+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~C3RPQGBD81L1.0 size=1048576 2023-12-03T11:30:43.195 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.194+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~C3RPQGBD81L1.1 size=307187 2023-12-03T11:30:43.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.198+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=C3RPQGBD81L1 tag=8DFRBB90BOQD type=write size=2404339 2023-12-03T11:30:43.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.200+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~120OEQVXZE5I.0 size=1048576 2023-12-03T11:30:43.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.201+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~120OEQVXZE5I.1 size=786509 2023-12-03T11:30:43.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.201+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=120OEQVXZE5I tag=TQP50XGDGN7M type=write size=2883661 2023-12-03T11:30:43.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.202+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.0 size=1048576 2023-12-03T11:30:43.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.203+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.1 size=1048576 2023-12-03T11:30:43.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.204+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~KZROCS408SS2.2 size=227182 2023-12-03T11:30:43.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.206+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=KZROCS408SS2 tag=18AQVNEY8UEB type=write size=3372910 2023-12-03T11:30:43.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.207+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=1RJPB90YMO8T type=del size=1932655 2023-12-03T11:30:43.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.210+0000 7fc553ded780 1 < suggested remove operation key=ZHK2M78A0SLQ tag=WF08UKNI4G2G type=del 2023-12-03T11:30:43.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.211+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=120OEHFTHY8B type=del size=2560134 2023-12-03T11:30:43.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.212+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.0 size=1048576 2023-12-03T11:30:43.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.213+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.1 size=1048576 2023-12-03T11:30:43.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.214+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~AUQMPWBE3JZL.2 size=873324 2023-12-03T11:30:43.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.216+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=AUQMPWBE3JZL tag=YHHPI9P8E5HC type=write size=4019052 2023-12-03T11:30:43.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.219+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.0 size=1048576 2023-12-03T11:30:43.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.220+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.1 size=1048576 2023-12-03T11:30:43.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.221+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.2 size=1002549 2023-12-03T11:30:43.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.221+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=ILNBTDJKLDMQ tag=E5OEVFJOQJUK type=write size=4148277 2023-12-03T11:30:43.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.221+0000 7fc553ded780 1 < canceled part key=_multipart_J664UUE697Q8.2~9JW1IXCE9H46.0 size=1048576 2023-12-03T11:30:43.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.222+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~9JW1IXCE9H46.1 size=45150 2023-12-03T11:30:43.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.224+0000 7fc553ded780 1 < canceled multipart upload key=J664UUE697Q8 upload=9JW1IXCE9H46 tag=BJDPBO2LW3LZ type=write size=2142302 2023-12-03T11:30:43.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.225+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=8RNKITF7QVWD type=del size=2553164 2023-12-03T11:30:43.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.227+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=6IM8J2FK6CAL type=del 2023-12-03T11:30:43.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.228+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=GWLN1YDH9AXQ type=write size=864249 2023-12-03T11:30:43.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.229+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.0 size=1048576 2023-12-03T11:30:43.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.229+0000 7fc553ded780 1 < canceled part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.1 size=1048576 2023-12-03T11:30:43.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.230+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~HJRSV90BSXHS.2 size=735710 2023-12-03T11:30:43.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.232+0000 7fc553ded780 1 < canceled multipart upload key=JUYA4LXEQ4US upload=HJRSV90BSXHS tag=OKOYXEC7QSNQ type=write size=3881438 2023-12-03T11:30:43.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.233+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=AP0NDMH9SZS7 type=del size=1164593 2023-12-03T11:30:43.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.234+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=LN7W3TT0A8J5 type=write size=320564 2023-12-03T11:30:43.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.237+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~VM92AUIZTT24.0 size=206280 2023-12-03T11:30:43.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.237+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=VM92AUIZTT24 tag=FT48AYVNOVWE type=write size=1254856 2023-12-03T11:30:43.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.238+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=Q27H4IP5A7DN type=del 2023-12-03T11:30:43.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.239+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=TNMKMMWIGG43 type=del size=1707926 2023-12-03T11:30:43.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.240+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~1FTIHPQO8E2D.0 size=762636 2023-12-03T11:30:43.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.242+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=1FTIHPQO8E2D tag=30VR6PJ8HXS2 type=write size=1811212 2023-12-03T11:30:43.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.245+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~9V9GMDFBDAJ5.0 size=75074 2023-12-03T11:30:43.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.245+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=9V9GMDFBDAJ5 tag=M0CFHTVLJY2X type=write size=1123650 2023-12-03T11:30:43.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.246+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.0 size=1048576 2023-12-03T11:30:43.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.247+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.1 size=1048576 2023-12-03T11:30:43.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.248+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~5U3J4RR00AGP.2 size=793229 2023-12-03T11:30:43.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.250+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=5U3J4RR00AGP tag=1YNHQ7VKWK17 type=write size=3938957 2023-12-03T11:30:43.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.251+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=TSHTXVM9M722 type=write size=1036983 2023-12-03T11:30:43.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.252+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~C3KI7G1NGYJF.0 size=918692 2023-12-03T11:30:43.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.254+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=C3KI7G1NGYJF tag=EO0HRD7B5Y6X type=write size=1967268 2023-12-03T11:30:43.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.256+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=4MYSKXMJAEQL type=del size=2482700 2023-12-03T11:30:43.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.257+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~A0JE3U18Z28E.0 size=1048576 2023-12-03T11:30:43.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.258+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~A0JE3U18Z28E.1 size=476054 2023-12-03T11:30:43.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.260+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=A0JE3U18Z28E tag=JWZZUYBR95AD type=write size=2573206 2023-12-03T11:30:43.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.262+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~ERS908U2GER3.0 size=17945 2023-12-03T11:30:43.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.262+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=ERS908U2GER3 tag=JTBLVPTYSE3Z type=write size=1066521 2023-12-03T11:30:43.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.263+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.0 size=1048576 2023-12-03T11:30:43.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.264+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.1 size=1048576 2023-12-03T11:30:43.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.265+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~VKK5MJVBWAFP.2 size=1046405 2023-12-03T11:30:43.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.268+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=VKK5MJVBWAFP tag=GIHUMMNDZX2V type=write size=4192133 2023-12-03T11:30:43.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.269+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=LP2NKO9L36DW type=del size=3869108 2023-12-03T11:30:43.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.270+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=PLIOA8EX81PV type=del 2023-12-03T11:30:43.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.271+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=FRT1F9EESE5E type=write size=419269 2023-12-03T11:30:43.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.272+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=AN0PVHHIRHHO type=del 2023-12-03T11:30:43.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.274+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=YYG45QMVE2RJ type=del size=3469885 2023-12-03T11:30:43.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.274+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~120OEQVXZE5I.0 size=1048576 2023-12-03T11:30:43.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.275+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~120OEQVXZE5I.1 size=786509 2023-12-03T11:30:43.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.277+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=120OEQVXZE5I tag=TQP50XGDGN7M type=write size=2883661 2023-12-03T11:30:43.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.279+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.0 size=1048576 2023-12-03T11:30:43.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.280+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.1 size=1048576 2023-12-03T11:30:43.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.281+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.2 size=620263 2023-12-03T11:30:43.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.281+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=3XTRJ7G4JYJC tag=0BEED2CXHAVU type=write size=3765991 2023-12-03T11:30:43.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.283+0000 7fc553ded780 1 < suggested remove operation key=0H5OZMBMLQFJ tag=1RJPB90YMO8T type=del 2023-12-03T11:30:43.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.284+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=746X3RDLJ35X type=del size=1041149 2023-12-03T11:30:43.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.286+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=120OEHFTHY8B type=del 2023-12-03T11:30:43.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.287+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=GR9Q3U2E4LGA type=write size=851525 2023-12-03T11:30:43.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.288+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.0 size=1048576 2023-12-03T11:30:43.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.289+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.1 size=1048576 2023-12-03T11:30:43.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.290+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~ILNBTDJKLDMQ.2 size=1002549 2023-12-03T11:30:43.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.292+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=ILNBTDJKLDMQ tag=E5OEVFJOQJUK type=write size=4148277 2023-12-03T11:30:43.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.294+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=3G4O4AXS6VTW type=del size=1688897 2023-12-03T11:30:43.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.295+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=8RNKITF7QVWD type=del 2023-12-03T11:30:43.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.297+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~DRT1EN81OU4E.0 size=1048576 2023-12-03T11:30:43.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.298+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~DRT1EN81OU4E.1 size=62124 2023-12-03T11:30:43.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.298+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=DRT1EN81OU4E tag=PUEN0UM4BFBO type=write size=2159276 2023-12-03T11:30:43.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.299+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=GWLN1YDH9AXQ type=write size=864249 2023-12-03T11:30:43.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.301+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=8GAXFTMO9CN5 type=del size=352818 2023-12-03T11:30:43.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.302+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=AP0NDMH9SZS7 type=del size=1164593 2023-12-03T11:30:43.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.304+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~YUOTLU2I9S58.0 size=1048576 2023-12-03T11:30:43.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.305+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~YUOTLU2I9S58.1 size=708297 2023-12-03T11:30:43.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.305+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=YUOTLU2I9S58 tag=3YFEVBSXPX5D type=write size=2805449 2023-12-03T11:30:43.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.306+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~VM92AUIZTT24.0 size=206280 2023-12-03T11:30:43.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.308+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=VM92AUIZTT24 tag=FT48AYVNOVWE type=write size=1254856 2023-12-03T11:30:43.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.311+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=I46TZPUNIE05 type=del size=3549145 2023-12-03T11:30:43.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.312+0000 7fc553ded780 1 < canceled operation key=IRMADF9MXXBZ tag=TNMKMMWIGG43 type=del size=1707926 2023-12-03T11:30:43.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.313+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=GT8G9OAEZMDS type=del size=2782047 2023-12-03T11:30:43.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.314+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~9V9GMDFBDAJ5.0 size=75074 2023-12-03T11:30:43.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.317+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=9V9GMDFBDAJ5 tag=M0CFHTVLJY2X type=write size=1123650 2023-12-03T11:30:43.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.318+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=EIY9KV81SAUH type=del size=2976119 2023-12-03T11:30:43.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.319+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=TSHTXVM9M722 type=write size=1036983 2023-12-03T11:30:43.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.322+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.0 size=1048576 2023-12-03T11:30:43.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.323+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.1 size=1048576 2023-12-03T11:30:43.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.324+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.2 size=707742 2023-12-03T11:30:43.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.324+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=PL6FRZFFD2SA tag=OUKIE6RN7MWG type=write size=3853470 2023-12-03T11:30:43.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.325+0000 7fc553ded780 1 < canceled operation key=FP2UZMDKYTD1 tag=4MYSKXMJAEQL type=del size=2482700 2023-12-03T11:30:43.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.327+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=HS2NSP2OJ4NN type=del size=34506 2023-12-03T11:30:43.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.328+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~ERS908U2GER3.0 size=17945 2023-12-03T11:30:43.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.331+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=ERS908U2GER3 tag=JTBLVPTYSE3Z type=write size=1066521 2023-12-03T11:30:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.333+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4O3V84OXJVJC.0 size=621715 2023-12-03T11:30:43.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.333+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=4O3V84OXJVJC tag=4MK9TST3C8A2 type=write size=1670291 2023-12-03T11:30:43.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.334+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=LP2NKO9L36DW type=del 2023-12-03T11:30:43.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.336+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.0 size=1048576 2023-12-03T11:30:43.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.337+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.1 size=1048576 2023-12-03T11:30:43.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.338+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.2 size=140220 2023-12-03T11:30:43.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.338+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=V7W0Z0JGMH3B tag=GB5C82EMFI79 type=write size=3285948 2023-12-03T11:30:43.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.339+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=FRT1F9EESE5E type=write size=419269 2023-12-03T11:30:43.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.340+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=53VD568RF1V1 type=del size=3513974 2023-12-03T11:30:43.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.342+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=YYG45QMVE2RJ type=del 2023-12-03T11:30:43.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.343+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=3E1OTFGE0YJ2 type=write size=921292 2023-12-03T11:30:43.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.344+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.0 size=1048576 2023-12-03T11:30:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.345+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.1 size=1048576 2023-12-03T11:30:43.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.345+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~3XTRJ7G4JYJC.2 size=620263 2023-12-03T11:30:43.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.347+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=3XTRJ7G4JYJC tag=0BEED2CXHAVU type=write size=3765991 2023-12-03T11:30:43.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.348+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=V5C20MG4HDSR type=del size=555194 2023-12-03T11:30:43.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.349+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=746X3RDLJ35X type=del 2023-12-03T11:30:43.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.350+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=X13PKGTHLTHX type=write size=274114 2023-12-03T11:30:43.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.353+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=GR9Q3U2E4LGA type=write 2023-12-03T11:30:43.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.354+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=3A7KIVNWWZUT type=write size=412585 2023-12-03T11:30:43.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.355+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=3G4O4AXS6VTW type=del size=1688897 2023-12-03T11:30:43.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.357+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.0 size=1048576 2023-12-03T11:30:43.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.358+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.1 size=1048576 2023-12-03T11:30:43.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.359+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.2 size=10912 2023-12-03T11:30:43.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.359+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=6RLZOGT72N2X tag=SNU2JKGHKOUG type=write size=3156640 2023-12-03T11:30:43.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.360+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~DRT1EN81OU4E.0 size=1048576 2023-12-03T11:30:43.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.361+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~DRT1EN81OU4E.1 size=62124 2023-12-03T11:30:43.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.364+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=DRT1EN81OU4E tag=PUEN0UM4BFBO type=write size=2159276 2023-12-03T11:30:43.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.366+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~IXDS6ZU1MRKU.0 size=624764 2023-12-03T11:30:43.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.366+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=IXDS6ZU1MRKU tag=ZOWB6IUG1NNM type=write size=1673340 2023-12-03T11:30:43.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.368+0000 7fc553ded780 1 < suggested update operation key=FP2UZMDKYTD1 tag=8GAXFTMO9CN5 type=del size=352818 2023-12-03T11:30:43.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.370+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.0 size=1048576 2023-12-03T11:30:43.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.371+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.1 size=1048576 2023-12-03T11:30:43.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.372+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.2 size=1039531 2023-12-03T11:30:43.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.372+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=B0BPL5L7CQRI tag=3BOLKI2OMJU0 type=write size=4185259 2023-12-03T11:30:43.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.372+0000 7fc553ded780 1 < canceled part key=_multipart_LRN5NYUZ5JT9.2~YUOTLU2I9S58.0 size=1048576 2023-12-03T11:30:43.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.373+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~YUOTLU2I9S58.1 size=708297 2023-12-03T11:30:43.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.375+0000 7fc553ded780 1 < canceled multipart upload key=LRN5NYUZ5JT9 upload=YUOTLU2I9S58 tag=3YFEVBSXPX5D type=write size=2805449 2023-12-03T11:30:43.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.377+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=LWG00IKBL0VO type=write size=620964 2023-12-03T11:30:43.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.378+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=I46TZPUNIE05 type=del 2023-12-03T11:30:43.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.380+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~C4KPI0ZQNUA3.0 size=1048576 2023-12-03T11:30:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.381+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~C4KPI0ZQNUA3.1 size=210800 2023-12-03T11:30:43.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.381+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=C4KPI0ZQNUA3 tag=HLQAKXN7NZEE type=write size=2307952 2023-12-03T11:30:43.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.383+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=GT8G9OAEZMDS type=del size=2782047 2023-12-03T11:30:43.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.385+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=Y9S5FX2CUH8M type=del size=389400 2023-12-03T11:30:43.387 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.386+0000 7fc553ded780 -1 < failed to complete operation key=706CET5PSDEG tag=EIY9KV81SAUH type=del size=2976119: (22) Invalid argument 2023-12-03T11:30:43.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.390+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~24UTH76SUFHH.0 size=109722 2023-12-03T11:30:43.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.390+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=24UTH76SUFHH tag=TWD4TEWIKP9H type=write size=1158298 2023-12-03T11:30:43.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.391+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.0 size=1048576 2023-12-03T11:30:43.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.391+0000 7fc553ded780 1 < canceled part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.1 size=1048576 2023-12-03T11:30:43.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.392+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PL6FRZFFD2SA.2 size=707742 2023-12-03T11:30:43.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.394+0000 7fc553ded780 1 < canceled multipart upload key=8GRA2MUBGWJI upload=PL6FRZFFD2SA tag=OUKIE6RN7MWG type=write size=3853470 2023-12-03T11:30:43.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.395+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=CDAE07GMROGQ type=write size=886640 2023-12-03T11:30:43.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.396+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=HS2NSP2OJ4NN type=del 2023-12-03T11:30:43.398 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.397+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=GEV9YMES9H3C type=del size=1555817 2023-12-03T11:30:43.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.399+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~4O3V84OXJVJC.0 size=621715 2023-12-03T11:30:43.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.401+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=4O3V84OXJVJC tag=4MK9TST3C8A2 type=write size=1670291 2023-12-03T11:30:43.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.403+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.0 size=1048576 2023-12-03T11:30:43.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.404+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.1 size=1048576 2023-12-03T11:30:43.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.405+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.2 size=1000891 2023-12-03T11:30:43.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.405+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=VXA67UCOJ9PE tag=WQ20R3SHH4EH type=write size=4146619 2023-12-03T11:30:43.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.406+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.0 size=1048576 2023-12-03T11:30:43.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.407+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.1 size=1048576 2023-12-03T11:30:43.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.408+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~V7W0Z0JGMH3B.2 size=140220 2023-12-03T11:30:43.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.410+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=V7W0Z0JGMH3B tag=GB5C82EMFI79 type=write size=3285948 2023-12-03T11:30:43.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.413+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.0 size=1048576 2023-12-03T11:30:43.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.414+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.1 size=1048576 2023-12-03T11:30:43.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.415+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.2 size=380505 2023-12-03T11:30:43.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.415+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=8NTO4JFWCNTP tag=WYUT9XPUBXMO type=write size=3526233 2023-12-03T11:30:43.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.416+0000 7fc553ded780 -1 < failed to complete operation key=706CET5PSDEG tag=53VD568RF1V1 type=del size=3513974: (22) Invalid argument 2023-12-03T11:30:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.419+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.0 size=1048576 2023-12-03T11:30:43.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.420+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.1 size=1048576 2023-12-03T11:30:43.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.421+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.2 size=523990 2023-12-03T11:30:43.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.421+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=974N6XD3KNKG tag=Q7NBCVXFDSTU type=write size=3669718 2023-12-03T11:30:43.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.423+0000 7fc553ded780 1 < suggested remove operation key=YE73HBF64CS7 tag=3E1OTFGE0YJ2 type=write 2023-12-03T11:30:43.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.426+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~28MUD9NNAP7R.0 size=1048576 2023-12-03T11:30:43.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.427+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~28MUD9NNAP7R.1 size=901696 2023-12-03T11:30:43.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.427+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=28MUD9NNAP7R tag=UZ8EILQUWJU5 type=write size=2998848 2023-12-03T11:30:43.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.430+0000 7fc553ded780 1 < suggested remove operation key=WLTMDGRBF4C9 tag=V5C20MG4HDSR type=del 2023-12-03T11:30:43.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.431+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=O23E3GFPVZMV type=del size=2033796 2023-12-03T11:30:43.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.433+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=X13PKGTHLTHX type=write size=274114 2023-12-03T11:30:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.434+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=QAZ3B0AEXY5G type=write size=916954 2023-12-03T11:30:43.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.437+0000 7fc553ded780 1 < suggested update operation key=UKAL8FCVIRSH tag=3A7KIVNWWZUT type=write size=412585 2023-12-03T11:30:43.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.438+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=FR316PM7TNPH type=del size=2207806 2023-12-03T11:30:43.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.440+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.0 size=1048576 2023-12-03T11:30:43.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.441+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.1 size=1048576 2023-12-03T11:30:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.442+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~6RLZOGT72N2X.2 size=10912 2023-12-03T11:30:43.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.444+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=6RLZOGT72N2X tag=SNU2JKGHKOUG type=write size=3156640 2023-12-03T11:30:43.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.447+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~Y7A93J6WS14Q.0 size=565081 2023-12-03T11:30:43.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.447+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=Y7A93J6WS14Q tag=ABIV68Z7T5XM type=write size=1613657 2023-12-03T11:30:43.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.448+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~IXDS6ZU1MRKU.0 size=624764 2023-12-03T11:30:43.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.450+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=IXDS6ZU1MRKU tag=ZOWB6IUG1NNM type=write size=1673340 2023-12-03T11:30:43.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.451+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=A87K2DX8PAHM type=del size=2759010 2023-12-03T11:30:43.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.451+0000 7fc553ded780 1 < canceled part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.0 size=1048576 2023-12-03T11:30:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.452+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.1 size=1048576 2023-12-03T11:30:43.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.453+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~B0BPL5L7CQRI.2 size=1039531 2023-12-03T11:30:43.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.455+0000 7fc553ded780 1 < canceled multipart upload key=IRMADF9MXXBZ upload=B0BPL5L7CQRI tag=3BOLKI2OMJU0 type=write size=4185259 2023-12-03T11:30:43.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.456+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=PYEO6X71NRTK type=write size=57699 2023-12-03T11:30:43.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.458+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=LWG00IKBL0VO type=write size=620964 2023-12-03T11:30:43.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.460+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~03BBO09LA9L3.0 size=785438 2023-12-03T11:30:43.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.460+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=03BBO09LA9L3 tag=29W197KIL4PC type=write size=1834014 2023-12-03T11:30:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.462+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~C4KPI0ZQNUA3.0 size=1048576 2023-12-03T11:30:43.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.463+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~C4KPI0ZQNUA3.1 size=210800 2023-12-03T11:30:43.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.465+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=C4KPI0ZQNUA3 tag=HLQAKXN7NZEE type=write size=2307952 2023-12-03T11:30:43.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.466+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=VYYTKEL8XIMX type=write size=453412 2023-12-03T11:30:43.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.467+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=Y9S5FX2CUH8M type=del 2023-12-03T11:30:43.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.469+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~H49AX4XJ4C0Y.0 size=1048576 2023-12-03T11:30:43.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.470+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~H49AX4XJ4C0Y.1 size=687428 2023-12-03T11:30:43.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.470+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=H49AX4XJ4C0Y tag=LAIF6QAP0QO7 type=write size=2784580 2023-12-03T11:30:43.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.472+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~24UTH76SUFHH.0 size=109722 2023-12-03T11:30:43.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.474+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=24UTH76SUFHH tag=TWD4TEWIKP9H type=write size=1158298 2023-12-03T11:30:43.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.475+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=YE0HW7JJBYMQ type=del size=2887281 2023-12-03T11:30:43.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.476+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=CDAE07GMROGQ type=write size=886640 2023-12-03T11:30:43.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.478+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~7I20AL9MEWJT.0 size=351997 2023-12-03T11:30:43.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.478+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=7I20AL9MEWJT tag=QX3VI9O72UQG type=write size=1400573 2023-12-03T11:30:43.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.480+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=GEV9YMES9H3C type=del 2023-12-03T11:30:43.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.482+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~J8M3AS58I55J.0 size=816151 2023-12-03T11:30:43.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.482+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=J8M3AS58I55J tag=J4I52DLOCJGC type=write size=1864727 2023-12-03T11:30:43.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.483+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.0 size=1048576 2023-12-03T11:30:43.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.484+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.1 size=1048576 2023-12-03T11:30:43.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.485+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~VXA67UCOJ9PE.2 size=1000891 2023-12-03T11:30:43.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.488+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=VXA67UCOJ9PE tag=WQ20R3SHH4EH type=write size=4146619 2023-12-03T11:30:43.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.489+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=AXIEKQ01K36L type=del size=1816568 2023-12-03T11:30:43.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.490+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.0 size=1048576 2023-12-03T11:30:43.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.492+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.1 size=1048576 2023-12-03T11:30:43.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.493+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~8NTO4JFWCNTP.2 size=380505 2023-12-03T11:30:43.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.495+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=8NTO4JFWCNTP tag=WYUT9XPUBXMO type=write size=3526233 2023-12-03T11:30:43.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.497+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=K5ZDX33QJALY type=del size=1060365 2023-12-03T11:30:43.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.498+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.0 size=1048576 2023-12-03T11:30:43.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.499+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.1 size=1048576 2023-12-03T11:30:43.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.500+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~974N6XD3KNKG.2 size=523990 2023-12-03T11:30:43.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.503+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=974N6XD3KNKG tag=Q7NBCVXFDSTU type=write size=3669718 2023-12-03T11:30:43.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.504+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=0SN1H03SMI2X type=del size=3570117 2023-12-03T11:30:43.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.505+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~28MUD9NNAP7R.0 size=1048576 2023-12-03T11:30:43.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.506+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~28MUD9NNAP7R.1 size=901696 2023-12-03T11:30:43.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.508+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=28MUD9NNAP7R tag=UZ8EILQUWJU5 type=write size=2998848 2023-12-03T11:30:43.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.510+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=TBTTJGK7E7VO type=del size=1462977 2023-12-03T11:30:43.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.511+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=O23E3GFPVZMV type=del 2023-12-03T11:30:43.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.512+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=15IQFOK0KV5B type=write size=422191 2023-12-03T11:30:43.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.513+0000 7fc553ded780 1 < completed write operation key=WDV4CW863V33 tag=QAZ3B0AEXY5G type=write size=916954 2023-12-03T11:30:43.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.516+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~542JTG1ITBSY.0 size=312368 2023-12-03T11:30:43.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.516+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=542JTG1ITBSY tag=WOWVB4GOLAPD type=write size=1360944 2023-12-03T11:30:43.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.517+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=FR316PM7TNPH type=del 2023-12-03T11:30:43.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.519+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=MZNGOZSQ8Z4O type=del size=1927298 2023-12-03T11:30:43.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.520+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~Y7A93J6WS14Q.0 size=565081 2023-12-03T11:30:43.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.522+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=Y7A93J6WS14Q tag=ABIV68Z7T5XM type=write size=1613657 2023-12-03T11:30:43.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.523+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=29EY5CZ2JDB0 type=del size=2688307 2023-12-03T11:30:43.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.526+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=A87K2DX8PAHM type=del 2023-12-03T11:30:43.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.527+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=SCGZ5PTLZPWZ type=del size=1805058 2023-12-03T11:30:43.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.528+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=PYEO6X71NRTK type=write size=57699 2023-12-03T11:30:43.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.529+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=18PMTUPF65AV type=del size=2338237 2023-12-03T11:30:43.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.531+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~03BBO09LA9L3.0 size=785438 2023-12-03T11:30:43.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.533+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=03BBO09LA9L3 tag=29W197KIL4PC type=write size=1834014 2023-12-03T11:30:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.536+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~4I89I2O4L8VO.0 size=1048576 2023-12-03T11:30:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.537+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~4I89I2O4L8VO.1 size=743582 2023-12-03T11:30:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.537+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=4I89I2O4L8VO tag=MD2P8LP8ZFWL type=write size=2840734 2023-12-03T11:30:43.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.538+0000 7fc553ded780 1 < completed write operation key=FP2UZMDKYTD1 tag=VYYTKEL8XIMX type=write size=453412 2023-12-03T11:30:43.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.539+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=LRDGV2I3C01V type=del size=1942155 2023-12-03T11:30:43.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.541+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~H49AX4XJ4C0Y.0 size=1048576 2023-12-03T11:30:43.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.542+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~H49AX4XJ4C0Y.1 size=687428 2023-12-03T11:30:43.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.544+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=H49AX4XJ4C0Y tag=LAIF6QAP0QO7 type=write size=2784580 2023-12-03T11:30:43.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.547+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O9GTK25CYLDL.0 size=1048576 2023-12-03T11:30:43.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.548+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O9GTK25CYLDL.1 size=286373 2023-12-03T11:30:43.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.548+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=O9GTK25CYLDL tag=NVRDXMZVRC90 type=write size=2383525 2023-12-03T11:30:43.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.549+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=YE0HW7JJBYMQ type=del 2023-12-03T11:30:43.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.550+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=8TMPF20FUGAW type=write size=820616 2023-12-03T11:30:43.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.551+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~7I20AL9MEWJT.0 size=351997 2023-12-03T11:30:43.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.554+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=7I20AL9MEWJT tag=QX3VI9O72UQG type=write size=1400573 2023-12-03T11:30:43.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.556+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~A70QMPL11873.0 size=1048576 2023-12-03T11:30:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.557+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~A70QMPL11873.1 size=434709 2023-12-03T11:30:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.557+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=A70QMPL11873 tag=FS8EDBVJJUU6 type=write size=2531861 2023-12-03T11:30:43.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.558+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~J8M3AS58I55J.0 size=816151 2023-12-03T11:30:43.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.560+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=J8M3AS58I55J tag=J4I52DLOCJGC type=write size=1864727 2023-12-03T11:30:43.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.561+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=8UVW0L6M2CB3 type=write size=879740 2023-12-03T11:30:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.563+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=AXIEKQ01K36L type=del 2023-12-03T11:30:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.564+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=EJS2A6QWRZV8 type=write size=624384 2023-12-03T11:30:43.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.565+0000 7fc553ded780 1 < canceled operation key=JUDXD9DL4N43 tag=K5ZDX33QJALY type=del size=1060365 2023-12-03T11:30:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.568+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.0 size=1048576 2023-12-03T11:30:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.568+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.1 size=1048576 2023-12-03T11:30:43.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.570+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.2 size=969870 2023-12-03T11:30:43.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.570+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=98NRK1K2MGHA tag=DZYIX0SAL4M9 type=write size=4115598 2023-12-03T11:30:43.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.571+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=0SN1H03SMI2X type=del 2023-12-03T11:30:43.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.573+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~HO23GX4M6UKF.0 size=332946 2023-12-03T11:30:43.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.573+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=HO23GX4M6UKF tag=10KFUJHXUZUT type=write size=1381522 2023-12-03T11:30:43.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.574+0000 7fc553ded780 -1 < failed to complete operation key=PL22GPUFN3O0 tag=TBTTJGK7E7VO type=del size=1462977: (22) Invalid argument 2023-12-03T11:30:43.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.576+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=Q3CV3JT5JZ6E type=del size=48824 2023-12-03T11:30:43.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.578+0000 7fc553ded780 1 < suggested remove operation key=ICDF1EYNI5IU tag=15IQFOK0KV5B type=write 2023-12-03T11:30:43.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.581+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.0 size=1048576 2023-12-03T11:30:43.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.582+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.1 size=1048576 2023-12-03T11:30:43.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.583+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.2 size=190858 2023-12-03T11:30:43.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.583+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=7RSK89S2Q1V8 tag=LLOKEA9RHQB2 type=write size=3336586 2023-12-03T11:30:43.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.584+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~542JTG1ITBSY.0 size=312368 2023-12-03T11:30:43.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.586+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=542JTG1ITBSY tag=WOWVB4GOLAPD type=write size=1360944 2023-12-03T11:30:43.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.588+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.0 size=1048576 2023-12-03T11:30:43.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.589+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.1 size=1048576 2023-12-03T11:30:43.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.590+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.2 size=214336 2023-12-03T11:30:43.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.590+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=P4T0LRUA6HK9 tag=LL5C68N9F7V1 type=write size=3360064 2023-12-03T11:30:43.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.592+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=MZNGOZSQ8Z4O type=del 2023-12-03T11:30:43.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.593+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=OF47M0KSZ5V3 type=del size=180440 2023-12-03T11:30:43.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.594+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=29EY5CZ2JDB0 type=del 2023-12-03T11:30:43.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.595+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=G4YND27AWNVO type=del size=2459034 2023-12-03T11:30:43.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.596+0000 7fc553ded780 -1 < failed to complete operation key=ICDF1EYNI5IU tag=SCGZ5PTLZPWZ type=del size=1805058: (22) Invalid argument 2023-12-03T11:30:43.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.598+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=E1GT5A1W51JP type=del size=2690063 2023-12-03T11:30:43.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.599+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=18PMTUPF65AV type=del 2023-12-03T11:30:43.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.600+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=QSWNDHIR0RFV type=del size=2712063 2023-12-03T11:30:43.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.601+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~4I89I2O4L8VO.0 size=1048576 2023-12-03T11:30:43.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.602+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~4I89I2O4L8VO.1 size=743582 2023-12-03T11:30:43.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.604+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=4I89I2O4L8VO tag=MD2P8LP8ZFWL type=write size=2840734 2023-12-03T11:30:43.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.606+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~LTY5Y5375U21.0 size=339756 2023-12-03T11:30:43.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.606+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=LTY5Y5375U21 tag=M6XE7WFA3W9J type=write size=1388332 2023-12-03T11:30:43.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.607+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=LRDGV2I3C01V type=del 2023-12-03T11:30:43.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.610+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~39L9CO3WO99B.0 size=1048576 2023-12-03T11:30:43.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.611+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~39L9CO3WO99B.1 size=96825 2023-12-03T11:30:43.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.611+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=39L9CO3WO99B tag=RTN9GBACLTLG type=write size=2193977 2023-12-03T11:30:43.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.612+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~O9GTK25CYLDL.0 size=1048576 2023-12-03T11:30:43.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.613+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~O9GTK25CYLDL.1 size=286373 2023-12-03T11:30:43.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.615+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=O9GTK25CYLDL tag=NVRDXMZVRC90 type=write size=2383525 2023-12-03T11:30:43.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.616+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=OXVBT2N8NR43 type=del size=629531 2023-12-03T11:30:43.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.617+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=8TMPF20FUGAW type=write size=820616 2023-12-03T11:30:43.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.618+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=EVHR363OGPWK type=del size=3156846 2023-12-03T11:30:43.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.619+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~A70QMPL11873.0 size=1048576 2023-12-03T11:30:43.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.619+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~A70QMPL11873.1 size=434709 2023-12-03T11:30:43.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.621+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=A70QMPL11873 tag=FS8EDBVJJUU6 type=write size=2531861 2023-12-03T11:30:43.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.623+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~S6WAI5VCTW5Q.0 size=188090 2023-12-03T11:30:43.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.623+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=S6WAI5VCTW5Q tag=MRFENH270ELA type=write size=1236666 2023-12-03T11:30:43.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.624+0000 7fc553ded780 1 < completed write operation key=FP2UZMDKYTD1 tag=8UVW0L6M2CB3 type=write size=879740 2023-12-03T11:30:43.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.625+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=NAAH26C30TWZ type=del size=3034321 2023-12-03T11:30:43.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.627+0000 7fc553ded780 1 < canceled operation key=ZUNOFY5N3J3H tag=EJS2A6QWRZV8 type=write size=624384 2023-12-03T11:30:43.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.629+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~66EAG4D3F4HQ.0 size=7878 2023-12-03T11:30:43.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.629+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=66EAG4D3F4HQ tag=QGAAHF2C5YTV type=write size=1056454 2023-12-03T11:30:43.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.629+0000 7fc553ded780 1 < canceled part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.0 size=1048576 2023-12-03T11:30:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.630+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.1 size=1048576 2023-12-03T11:30:43.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.631+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~98NRK1K2MGHA.2 size=969870 2023-12-03T11:30:43.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.633+0000 7fc553ded780 1 < canceled multipart upload key=JUYA4LXEQ4US upload=98NRK1K2MGHA tag=DZYIX0SAL4M9 type=write size=4115598 2023-12-03T11:30:43.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.635+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=DXG2K29USTJO type=del size=3605708 2023-12-03T11:30:43.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.636+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~HO23GX4M6UKF.0 size=332946 2023-12-03T11:30:43.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.638+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=HO23GX4M6UKF tag=10KFUJHXUZUT type=write size=1381522 2023-12-03T11:30:43.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.639+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=0V8KF8ZTC1ID type=del size=3853064 2023-12-03T11:30:43.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.641+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=Q3CV3JT5JZ6E type=del 2023-12-03T11:30:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.642+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=DXLYZZ02GGGD type=write size=856231 2023-12-03T11:30:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.643+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.0 size=1048576 2023-12-03T11:30:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.644+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.1 size=1048576 2023-12-03T11:30:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.645+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~7RSK89S2Q1V8.2 size=190858 2023-12-03T11:30:43.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.647+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=7RSK89S2Q1V8 tag=LLOKEA9RHQB2 type=write size=3336586 2023-12-03T11:30:43.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.649+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=REBCQQB47DGI type=write size=698376 2023-12-03T11:30:43.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.650+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.0 size=1048576 2023-12-03T11:30:43.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.651+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.1 size=1048576 2023-12-03T11:30:43.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.652+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~P4T0LRUA6HK9.2 size=214336 2023-12-03T11:30:43.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.654+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=P4T0LRUA6HK9 tag=LL5C68N9F7V1 type=write size=3360064 2023-12-03T11:30:43.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.657+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~02UC1VKXKHJK.0 size=683919 2023-12-03T11:30:43.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.657+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=02UC1VKXKHJK tag=09XMYAJG1X2L type=write size=1732495 2023-12-03T11:30:43.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.658+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=OF47M0KSZ5V3 type=del 2023-12-03T11:30:43.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.661+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~4T0FUK9YY1NX.0 size=1048576 2023-12-03T11:30:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.664+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~4T0FUK9YY1NX.1 size=128843 2023-12-03T11:30:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.664+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=4T0FUK9YY1NX tag=1RD1QL88S2YA type=write size=2225995 2023-12-03T11:30:43.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.667+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=G4YND27AWNVO type=del 2023-12-03T11:30:43.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.669+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.0 size=1048576 2023-12-03T11:30:43.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.670+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.1 size=1048576 2023-12-03T11:30:43.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.672+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.2 size=472447 2023-12-03T11:30:43.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.672+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=3CCZL6ZKNR2G tag=2Z9342TR6773 type=write size=3618175 2023-12-03T11:30:43.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.674+0000 7fc553ded780 1 < canceled operation key=J664UUE697Q8 tag=E1GT5A1W51JP type=del size=2690063 2023-12-03T11:30:43.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.675+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=JPYLACWCKKX9 type=del size=1376318 2023-12-03T11:30:43.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.677+0000 7fc553ded780 1 < suggested remove operation key=6OW429S51UWN tag=QSWNDHIR0RFV type=del 2023-12-03T11:30:43.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.678+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=1SYF74DYQI43 type=del size=1012814 2023-12-03T11:30:43.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.680+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~LTY5Y5375U21.0 size=339756 2023-12-03T11:30:43.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.681+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=LTY5Y5375U21 tag=M6XE7WFA3W9J type=write size=1388332 2023-12-03T11:30:43.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.683+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=RQQ11F4SCYLX type=del size=42011 2023-12-03T11:30:43.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.683+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~39L9CO3WO99B.0 size=1048576 2023-12-03T11:30:43.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.684+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~39L9CO3WO99B.1 size=96825 2023-12-03T11:30:43.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.686+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=39L9CO3WO99B tag=RTN9GBACLTLG type=write size=2193977 2023-12-03T11:30:43.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.689+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.0 size=1048576 2023-12-03T11:30:43.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.690+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.1 size=1048576 2023-12-03T11:30:43.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.691+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.2 size=716780 2023-12-03T11:30:43.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.691+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=IWXXX5079EW7 tag=DLIWY6J15MEJ type=write size=3862508 2023-12-03T11:30:43.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.695+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=OXVBT2N8NR43 type=del 2023-12-03T11:30:43.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.697+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~5XCOEJ0A7VIR.0 size=1048576 2023-12-03T11:30:43.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.698+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~5XCOEJ0A7VIR.1 size=664899 2023-12-03T11:30:43.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.698+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=5XCOEJ0A7VIR tag=URSMZOJJYBYL type=write size=2762051 2023-12-03T11:30:43.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.701+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=EVHR363OGPWK type=del size=3156846 2023-12-03T11:30:43.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.702+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=JJ6WZZ5UI7WL type=del size=1633962 2023-12-03T11:30:43.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.703+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~S6WAI5VCTW5Q.0 size=188090 2023-12-03T11:30:43.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.706+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=S6WAI5VCTW5Q tag=MRFENH270ELA type=write size=1236666 2023-12-03T11:30:43.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.707+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=DQTXMSP8VIHR type=del size=613168 2023-12-03T11:30:43.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.710+0000 7fc553ded780 1 < suggested remove operation key=TQRZT37AB9WC tag=NAAH26C30TWZ type=del 2023-12-03T11:30:43.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.711+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=WWV4PQMZHX4A type=del size=649550 2023-12-03T11:30:43.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.713+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~66EAG4D3F4HQ.0 size=7878 2023-12-03T11:30:43.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.715+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=66EAG4D3F4HQ tag=QGAAHF2C5YTV type=write size=1056454 2023-12-03T11:30:43.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.716+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=ZR720VYMA0GX type=del size=1729811 2023-12-03T11:30:43.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.717+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=DXG2K29USTJO type=del 2023-12-03T11:30:43.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.719+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=20WQ0FDIPUXU type=del size=2090040 2023-12-03T11:30:43.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.720+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=0V8KF8ZTC1ID type=del 2023-12-03T11:30:43.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.722+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0CO1TQEDWSQJ.0 size=1048576 2023-12-03T11:30:43.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.723+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0CO1TQEDWSQJ.1 size=324205 2023-12-03T11:30:43.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.723+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=0CO1TQEDWSQJ tag=QLR5A4XPCUH7 type=write size=2421357 2023-12-03T11:30:43.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.725+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=DXLYZZ02GGGD type=write size=856231 2023-12-03T11:30:43.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.726+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=HOUDE2Q7TL6C type=del size=557173 2023-12-03T11:30:43.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.729+0000 7fc553ded780 1 < suggested remove operation key=ZUNOFY5N3J3H tag=REBCQQB47DGI type=write 2023-12-03T11:30:43.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.730+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=NKP3C2PGGKNY type=del size=1624761 2023-12-03T11:30:43.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.731+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~02UC1VKXKHJK.0 size=683919 2023-12-03T11:30:43.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.733+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=02UC1VKXKHJK tag=09XMYAJG1X2L type=write size=1732495 2023-12-03T11:30:43.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.736+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.0 size=1048576 2023-12-03T11:30:43.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.737+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.1 size=1048576 2023-12-03T11:30:43.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.738+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.2 size=177180 2023-12-03T11:30:43.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.738+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=EK7QPPIG99I8 tag=LTMSXW7PVU76 type=write size=3322908 2023-12-03T11:30:43.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.739+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~4T0FUK9YY1NX.0 size=1048576 2023-12-03T11:30:43.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.740+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~4T0FUK9YY1NX.1 size=128843 2023-12-03T11:30:43.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.743+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=4T0FUK9YY1NX tag=1RD1QL88S2YA type=write size=2225995 2023-12-03T11:30:43.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.746+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.0 size=1048576 2023-12-03T11:30:43.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.747+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.1 size=1048576 2023-12-03T11:30:43.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.748+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.2 size=846933 2023-12-03T11:30:43.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.748+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=JYEW4FDZM744 tag=Z456YEFULPWR type=write size=3992661 2023-12-03T11:30:43.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.749+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.0 size=1048576 2023-12-03T11:30:43.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.750+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.1 size=1048576 2023-12-03T11:30:43.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.751+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~3CCZL6ZKNR2G.2 size=472447 2023-12-03T11:30:43.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.754+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=3CCZL6ZKNR2G tag=2Z9342TR6773 type=write size=3618175 2023-12-03T11:30:43.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.755+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=IMVGGLU513HK type=del size=102693 2023-12-03T11:30:43.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.756+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=JPYLACWCKKX9 type=del 2023-12-03T11:30:43.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.757+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=CJD0SLSW5WME type=del size=947148 2023-12-03T11:30:43.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.759+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=1SYF74DYQI43 type=del 2023-12-03T11:30:43.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.761+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.0 size=1048576 2023-12-03T11:30:43.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.762+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.1 size=1048576 2023-12-03T11:30:43.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.763+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.2 size=819814 2023-12-03T11:30:43.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.763+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=94WEM5KA43DZ tag=DGV52C2QJ2D7 type=write size=3965542 2023-12-03T11:30:43.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.765+0000 7fc553ded780 1 < suggested remove operation key=ZHK2M78A0SLQ tag=RQQ11F4SCYLX type=del 2023-12-03T11:30:43.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.768+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~KJHJEIWFKC5R.0 size=1048576 2023-12-03T11:30:43.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.769+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~KJHJEIWFKC5R.1 size=970104 2023-12-03T11:30:43.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.769+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=KJHJEIWFKC5R tag=S4NR91S0O8CA type=write size=3067256 2023-12-03T11:30:43.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.771+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.0 size=1048576 2023-12-03T11:30:43.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.771+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.1 size=1048576 2023-12-03T11:30:43.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.772+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~IWXXX5079EW7.2 size=716780 2023-12-03T11:30:43.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.774+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=IWXXX5079EW7 tag=DLIWY6J15MEJ type=write size=3862508 2023-12-03T11:30:43.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.777+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~PYFOSUF7VROR.0 size=1048576 2023-12-03T11:30:43.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.778+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~PYFOSUF7VROR.1 size=799885 2023-12-03T11:30:43.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.778+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=PYFOSUF7VROR tag=FVGG0KCH8LPX type=write size=2897037 2023-12-03T11:30:43.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.778+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~5XCOEJ0A7VIR.0 size=1048576 2023-12-03T11:30:43.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.778+0000 7fc553ded780 1 < canceled part key=_multipart_JUDXD9DL4N43.2~5XCOEJ0A7VIR.1 size=664899 2023-12-03T11:30:43.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.780+0000 7fc553ded780 1 < canceled multipart upload key=JUDXD9DL4N43 upload=5XCOEJ0A7VIR tag=URSMZOJJYBYL type=write size=2762051 2023-12-03T11:30:43.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.781+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=9JCLY6BXCPQX type=del size=2317782 2023-12-03T11:30:43.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.782+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=JJ6WZZ5UI7WL type=del 2023-12-03T11:30:43.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.785+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~C19CFWOV3CR5.0 size=1048576 2023-12-03T11:30:43.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.786+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~C19CFWOV3CR5.1 size=459401 2023-12-03T11:30:43.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.786+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=C19CFWOV3CR5 tag=CW4NEGBGWIW5 type=write size=2556553 2023-12-03T11:30:43.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.788+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=DQTXMSP8VIHR type=del 2023-12-03T11:30:43.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.789+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=VJ88YO7UKE6Q type=del size=414736 2023-12-03T11:30:43.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.791+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=WWV4PQMZHX4A type=del 2023-12-03T11:30:43.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.792+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=UFBBWKKP86DZ type=del size=61463 2023-12-03T11:30:43.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.793+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=ZR720VYMA0GX type=del 2023-12-03T11:30:43.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.794+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=2BICWQP2CLKU type=write size=267016 2023-12-03T11:30:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.795+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=20WQ0FDIPUXU type=del 2023-12-03T11:30:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.797+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=089YA3DETUW6 type=write size=262883 2023-12-03T11:30:43.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.798+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0CO1TQEDWSQJ.0 size=1048576 2023-12-03T11:30:43.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.799+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0CO1TQEDWSQJ.1 size=324205 2023-12-03T11:30:43.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.801+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=0CO1TQEDWSQJ tag=QLR5A4XPCUH7 type=write size=2421357 2023-12-03T11:30:43.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.802+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=B7RFFDRIOCWG type=write size=1044925 2023-12-03T11:30:43.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.803+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=HOUDE2Q7TL6C type=del 2023-12-03T11:30:43.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.804+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=BWH5V2AJUSB9 type=del size=2786473 2023-12-03T11:30:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.806+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=NKP3C2PGGKNY type=del 2023-12-03T11:30:43.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.808+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~ION033TGR0Y5.0 size=359809 2023-12-03T11:30:43.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.808+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=ION033TGR0Y5 tag=F9A5XK559XV5 type=write size=1408385 2023-12-03T11:30:43.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.809+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.0 size=1048576 2023-12-03T11:30:43.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.810+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.1 size=1048576 2023-12-03T11:30:43.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.811+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~EK7QPPIG99I8.2 size=177180 2023-12-03T11:30:43.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.813+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=EK7QPPIG99I8 tag=LTMSXW7PVU76 type=write size=3322908 2023-12-03T11:30:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.815+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~76DINM925QG9.0 size=288167 2023-12-03T11:30:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.815+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=76DINM925QG9 tag=3MCAJ6V0HA6N type=write size=1336743 2023-12-03T11:30:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.816+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.0 size=1048576 2023-12-03T11:30:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.817+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.1 size=1048576 2023-12-03T11:30:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.818+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~JYEW4FDZM744.2 size=846933 2023-12-03T11:30:43.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.820+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=JYEW4FDZM744 tag=Z456YEFULPWR type=write size=3992661 2023-12-03T11:30:43.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.822+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~4JK1K07F44UU.0 size=151201 2023-12-03T11:30:43.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.822+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=4JK1K07F44UU tag=QHG4N1CFJLII type=write size=1199777 2023-12-03T11:30:43.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.824+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=IMVGGLU513HK type=del 2023-12-03T11:30:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.826+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.0 size=1048576 2023-12-03T11:30:43.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.828+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.1 size=1048576 2023-12-03T11:30:43.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.829+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.2 size=132031 2023-12-03T11:30:43.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.829+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=YPUSGCA8SOX6 tag=B1ASASSU6Z9F type=write size=3277759 2023-12-03T11:30:43.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.830+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=CJD0SLSW5WME type=del 2023-12-03T11:30:43.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.832+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~GIJLPE6PY018.0 size=1048576 2023-12-03T11:30:43.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.833+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~GIJLPE6PY018.1 size=75223 2023-12-03T11:30:43.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.833+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=GIJLPE6PY018 tag=WHP5APNCH27H type=write size=2172375 2023-12-03T11:30:43.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.835+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.0 size=1048576 2023-12-03T11:30:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.836+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.1 size=1048576 2023-12-03T11:30:43.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.837+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~94WEM5KA43DZ.2 size=819814 2023-12-03T11:30:43.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.839+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=94WEM5KA43DZ tag=DGV52C2QJ2D7 type=write size=3965542 2023-12-03T11:30:43.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.840+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=DV2Z32F24IDT type=del size=1499240 2023-12-03T11:30:43.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.841+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~KJHJEIWFKC5R.0 size=1048576 2023-12-03T11:30:43.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.842+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~KJHJEIWFKC5R.1 size=970104 2023-12-03T11:30:43.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.844+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=KJHJEIWFKC5R tag=S4NR91S0O8CA type=write size=3067256 2023-12-03T11:30:43.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.847+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~YCH0YAKRH5EI.0 size=1048576 2023-12-03T11:30:43.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.847+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~YCH0YAKRH5EI.1 size=695140 2023-12-03T11:30:43.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.847+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=YCH0YAKRH5EI tag=WMM54XYQ1NWG type=write size=2792292 2023-12-03T11:30:43.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.849+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~PYFOSUF7VROR.0 size=1048576 2023-12-03T11:30:43.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.850+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~PYFOSUF7VROR.1 size=799885 2023-12-03T11:30:43.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.852+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=PYFOSUF7VROR tag=FVGG0KCH8LPX type=write size=2897037 2023-12-03T11:30:43.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.853+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=B8FY5MYXQ5IN type=del size=246469 2023-12-03T11:30:43.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.854+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=9JCLY6BXCPQX type=del 2023-12-03T11:30:43.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.855+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=WYB2TG8K6C13 type=del size=3987065 2023-12-03T11:30:43.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.857+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~C19CFWOV3CR5.0 size=1048576 2023-12-03T11:30:43.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.858+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~C19CFWOV3CR5.1 size=459401 2023-12-03T11:30:43.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.860+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=C19CFWOV3CR5 tag=CW4NEGBGWIW5 type=write size=2556553 2023-12-03T11:30:43.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.861+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=3OZUPQ14WQ2B type=del size=703263 2023-12-03T11:30:43.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.862+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=VJ88YO7UKE6Q type=del 2023-12-03T11:30:43.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.864+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.0 size=1048576 2023-12-03T11:30:43.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.865+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.1 size=1048576 2023-12-03T11:30:43.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.866+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.2 size=598598 2023-12-03T11:30:43.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.866+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=R66MALPGO6B4 tag=4HXB5XM31DOO type=write size=3744326 2023-12-03T11:30:43.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.867+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=UFBBWKKP86DZ type=del 2023-12-03T11:30:43.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.869+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=ZVF4X485VCQ2 type=del size=1517299 2023-12-03T11:30:43.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.870+0000 7fc553ded780 1 < canceled operation key=L4ME9RYRR4NG tag=2BICWQP2CLKU type=write size=267016 2023-12-03T11:30:43.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.872+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~8JQJGRIO3VA0.0 size=83521 2023-12-03T11:30:43.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.872+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=8JQJGRIO3VA0 tag=SDJ8HJ9PH5SY type=write size=1132097 2023-12-03T11:30:43.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.873+0000 7fc553ded780 1 < completed write operation key=WDV4CW863V33 tag=089YA3DETUW6 type=write size=262883 2023-12-03T11:30:43.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.875+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=M0KAYZ13AXHU type=del size=1736316 2023-12-03T11:30:43.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.876+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=B7RFFDRIOCWG type=write size=1044925 2023-12-03T11:30:43.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.878+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.0 size=1048576 2023-12-03T11:30:43.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.879+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.1 size=1048576 2023-12-03T11:30:43.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.880+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.2 size=310532 2023-12-03T11:30:43.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.880+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=1S819SOGEPRJ tag=ODP5Y6Z8YXBF type=write size=3456260 2023-12-03T11:30:43.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.881+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=BWH5V2AJUSB9 type=del 2023-12-03T11:30:43.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.884+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~001KN6Y7OTDZ.0 size=1048576 2023-12-03T11:30:43.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.885+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~001KN6Y7OTDZ.1 size=317714 2023-12-03T11:30:43.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.885+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=001KN6Y7OTDZ tag=MOQWZ3RY3XVH type=write size=2414866 2023-12-03T11:30:43.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.886+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~ION033TGR0Y5.0 size=359809 2023-12-03T11:30:43.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.888+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=ION033TGR0Y5 tag=F9A5XK559XV5 type=write size=1408385 2023-12-03T11:30:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.890+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~MZY7L0PIC8DB.0 size=839103 2023-12-03T11:30:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.890+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=MZY7L0PIC8DB tag=IIOC0FU0TKTU type=write size=1887679 2023-12-03T11:30:43.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.891+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~76DINM925QG9.0 size=288167 2023-12-03T11:30:43.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.893+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=76DINM925QG9 tag=3MCAJ6V0HA6N type=write size=1336743 2023-12-03T11:30:43.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.896+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~MH5COE9TSR13.0 size=1048576 2023-12-03T11:30:43.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.897+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~MH5COE9TSR13.1 size=698403 2023-12-03T11:30:43.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.897+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=MH5COE9TSR13 tag=N836JFABPPV2 type=write size=2795555 2023-12-03T11:30:43.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.898+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~4JK1K07F44UU.0 size=151201 2023-12-03T11:30:43.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.902+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=4JK1K07F44UU tag=QHG4N1CFJLII type=write size=1199777 2023-12-03T11:30:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.903+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=MBXFT1XICTLV type=del size=3774864 2023-12-03T11:30:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.904+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.0 size=1048576 2023-12-03T11:30:43.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.907+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.1 size=1048576 2023-12-03T11:30:43.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.908+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~YPUSGCA8SOX6.2 size=132031 2023-12-03T11:30:43.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.910+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=YPUSGCA8SOX6 tag=B1ASASSU6Z9F type=write size=3277759 2023-12-03T11:30:43.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.911+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=3GZUF88L0S1W type=del size=648184 2023-12-03T11:30:43.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.913+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~GIJLPE6PY018.0 size=1048576 2023-12-03T11:30:43.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.914+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~GIJLPE6PY018.1 size=75223 2023-12-03T11:30:43.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.916+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=GIJLPE6PY018 tag=WHP5APNCH27H type=write size=2172375 2023-12-03T11:30:43.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.917+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=2IIBNTXD53XJ type=del size=1320551 2023-12-03T11:30:43.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.920+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=DV2Z32F24IDT type=del 2023-12-03T11:30:43.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.921+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=H323VH2XXBJC type=write size=932805 2023-12-03T11:30:43.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.922+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~YCH0YAKRH5EI.0 size=1048576 2023-12-03T11:30:43.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.923+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~YCH0YAKRH5EI.1 size=695140 2023-12-03T11:30:43.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.926+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=YCH0YAKRH5EI tag=WMM54XYQ1NWG type=write size=2792292 2023-12-03T11:30:43.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.927+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=QBCO00F3J0UV type=del size=1141457 2023-12-03T11:30:43.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.928+0000 7fc553ded780 -1 < failed to complete operation key=IRMADF9MXXBZ tag=B8FY5MYXQ5IN type=del size=246469: (22) Invalid argument 2023-12-03T11:30:43.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.930+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~TM287SWX7MCK.0 size=1048576 2023-12-03T11:30:43.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.931+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~TM287SWX7MCK.1 size=722547 2023-12-03T11:30:43.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.931+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=TM287SWX7MCK tag=FOJJHPBP1T7A type=write size=2819699 2023-12-03T11:30:43.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.932+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=WYB2TG8K6C13 type=del 2023-12-03T11:30:43.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.935+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~NHS5Q6TIDW74.0 size=997540 2023-12-03T11:30:43.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.935+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=NHS5Q6TIDW74 tag=PH0H51TDLSD7 type=write size=2046116 2023-12-03T11:30:43.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.936+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=3OZUPQ14WQ2B type=del 2023-12-03T11:30:43.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.938+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~6B9QEGYUNA45.0 size=1048576 2023-12-03T11:30:43.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.940+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~6B9QEGYUNA45.1 size=859522 2023-12-03T11:30:43.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.940+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=6B9QEGYUNA45 tag=Z3SIM8S5Q09D type=write size=2956674 2023-12-03T11:30:43.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.941+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.0 size=1048576 2023-12-03T11:30:43.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.942+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.1 size=1048576 2023-12-03T11:30:43.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.943+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~R66MALPGO6B4.2 size=598598 2023-12-03T11:30:43.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.945+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=R66MALPGO6B4 tag=4HXB5XM31DOO type=write size=3744326 2023-12-03T11:30:43.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.948+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.0 size=1048576 2023-12-03T11:30:43.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.949+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.1 size=1048576 2023-12-03T11:30:43.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.949+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.2 size=325317 2023-12-03T11:30:43.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.949+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=86W1YH8I8E3E tag=6HUYUY51UW3A type=write size=3471045 2023-12-03T11:30:43.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.951+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=ZVF4X485VCQ2 type=del 2023-12-03T11:30:43.953 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.952+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=QM9USSP168XV type=del size=3437537 2023-12-03T11:30:43.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.953+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~8JQJGRIO3VA0.0 size=83521 2023-12-03T11:30:43.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.955+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=8JQJGRIO3VA0 tag=SDJ8HJ9PH5SY type=write size=1132097 2023-12-03T11:30:43.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.958+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S44EXREQYUJT.0 size=1048576 2023-12-03T11:30:43.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.959+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S44EXREQYUJT.1 size=120722 2023-12-03T11:30:43.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.959+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=S44EXREQYUJT tag=W4YZ404ENOPB type=write size=2217874 2023-12-03T11:30:43.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.960+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=M0KAYZ13AXHU type=del 2023-12-03T11:30:43.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.961+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=UDRO9FJ0JTIB type=del size=863889 2023-12-03T11:30:43.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.963+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.0 size=1048576 2023-12-03T11:30:43.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.964+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.1 size=1048576 2023-12-03T11:30:43.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.965+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~1S819SOGEPRJ.2 size=310532 2023-12-03T11:30:43.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.967+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=1S819SOGEPRJ tag=ODP5Y6Z8YXBF type=write size=3456260 2023-12-03T11:30:43.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.968+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=ZH47PG429MDT type=del size=214916 2023-12-03T11:30:43.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.969+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~001KN6Y7OTDZ.0 size=1048576 2023-12-03T11:30:43.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.970+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~001KN6Y7OTDZ.1 size=317714 2023-12-03T11:30:43.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.973+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=001KN6Y7OTDZ tag=MOQWZ3RY3XVH type=write size=2414866 2023-12-03T11:30:43.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.974+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=86JV74V3WZVZ type=write size=284333 2023-12-03T11:30:43.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.975+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~MZY7L0PIC8DB.0 size=839103 2023-12-03T11:30:43.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.977+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=MZY7L0PIC8DB tag=IIOC0FU0TKTU type=write size=1887679 2023-12-03T11:30:43.979 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.979+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=1OKTM9080G38 type=write size=805429 2023-12-03T11:30:43.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.980+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~MH5COE9TSR13.0 size=1048576 2023-12-03T11:30:43.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.981+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~MH5COE9TSR13.1 size=698403 2023-12-03T11:30:43.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.983+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=MH5COE9TSR13 tag=N836JFABPPV2 type=write size=2795555 2023-12-03T11:30:43.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.985+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.0 size=1048576 2023-12-03T11:30:43.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.986+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.1 size=1048576 2023-12-03T11:30:43.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.987+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.2 size=882327 2023-12-03T11:30:43.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.987+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=HWHDBE4VX6F2 tag=T8YPHUTZN2Z5 type=write size=4028055 2023-12-03T11:30:43.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.990+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=MBXFT1XICTLV type=del 2023-12-03T11:30:43.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.991+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=HX6L1MJ26ZIW type=del size=2380165 2023-12-03T11:30:43.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.993+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=3GZUF88L0S1W type=del 2023-12-03T11:30:43.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.994+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=89Q1R3CHK68I type=del size=2245263 2023-12-03T11:30:43.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.996+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=2IIBNTXD53XJ type=del 2023-12-03T11:30:43.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.998+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.0 size=1048576 2023-12-03T11:30:44.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:43.999+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.1 size=1048576 2023-12-03T11:30:44.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.001+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.2 size=956927 2023-12-03T11:30:44.002 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.001+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=TVQ16G5Q5L0S tag=K2XZCICA9RDA type=write size=4102655 2023-12-03T11:30:44.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.002+0000 7fc553ded780 1 < completed write operation key=FC4WU6SMNEWV tag=H323VH2XXBJC type=write size=932805 2023-12-03T11:30:44.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.003+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=7DQ0GF9RT6N4 type=del size=460171 2023-12-03T11:30:44.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.005+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=QBCO00F3J0UV type=del 2023-12-03T11:30:44.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.006+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=PHIVIPWIWTS9 type=del size=3002531 2023-12-03T11:30:44.009 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.008+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~TM287SWX7MCK.0 size=1048576 2023-12-03T11:30:44.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.009+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~TM287SWX7MCK.1 size=722547 2023-12-03T11:30:44.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.012+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=TM287SWX7MCK tag=FOJJHPBP1T7A type=write size=2819699 2023-12-03T11:30:44.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.014+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~J26PJJN29MZM.0 size=1048576 2023-12-03T11:30:44.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.015+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~J26PJJN29MZM.1 size=689774 2023-12-03T11:30:44.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.015+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=J26PJJN29MZM tag=UV963H09JDKX type=write size=2786926 2023-12-03T11:30:44.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.016+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~NHS5Q6TIDW74.0 size=997540 2023-12-03T11:30:44.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.018+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=NHS5Q6TIDW74 tag=PH0H51TDLSD7 type=write size=2046116 2023-12-03T11:30:44.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.020+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~16LZ1BB0JTIO.0 size=1048576 2023-12-03T11:30:44.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.022+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~16LZ1BB0JTIO.1 size=455072 2023-12-03T11:30:44.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.022+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=16LZ1BB0JTIO tag=CA48AJPROFPE type=write size=2552224 2023-12-03T11:30:44.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.022+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~6B9QEGYUNA45.0 size=1048576 2023-12-03T11:30:44.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.023+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~6B9QEGYUNA45.1 size=859522 2023-12-03T11:30:44.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.025+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=6B9QEGYUNA45 tag=Z3SIM8S5Q09D type=write size=2956674 2023-12-03T11:30:44.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.028+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.0 size=1048576 2023-12-03T11:30:44.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.029+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.1 size=1048576 2023-12-03T11:30:44.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.029+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.2 size=349877 2023-12-03T11:30:44.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.029+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=B45YET0BKGCN tag=NNQWM2JQSGHG type=write size=3495605 2023-12-03T11:30:44.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.031+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.0 size=1048576 2023-12-03T11:30:44.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.032+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.1 size=1048576 2023-12-03T11:30:44.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.033+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~86W1YH8I8E3E.2 size=325317 2023-12-03T11:30:44.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.036+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=86W1YH8I8E3E tag=6HUYUY51UW3A type=write size=3471045 2023-12-03T11:30:44.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.038+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~D2ALGUQ1MYOU.0 size=887284 2023-12-03T11:30:44.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.038+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=D2ALGUQ1MYOU tag=WJ0ZR61HTTDJ type=write size=1935860 2023-12-03T11:30:44.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.039+0000 7fc553ded780 1 < canceled operation key=TQRZT37AB9WC tag=QM9USSP168XV type=del size=3437537 2023-12-03T11:30:44.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.041+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~3UNAZLK1Z6LT.0 size=1048576 2023-12-03T11:30:44.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.042+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~3UNAZLK1Z6LT.1 size=8600 2023-12-03T11:30:44.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.042+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=3UNAZLK1Z6LT tag=PN6Z4892SC7L type=write size=2105752 2023-12-03T11:30:44.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.044+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S44EXREQYUJT.0 size=1048576 2023-12-03T11:30:44.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.045+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S44EXREQYUJT.1 size=120722 2023-12-03T11:30:44.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.047+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=S44EXREQYUJT tag=W4YZ404ENOPB type=write size=2217874 2023-12-03T11:30:44.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.048+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=HY1MEZCFNJFP type=del size=256225 2023-12-03T11:30:44.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.049+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=UDRO9FJ0JTIB type=del 2023-12-03T11:30:44.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.051+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~8Z7POSZW0485.0 size=562881 2023-12-03T11:30:44.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.051+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=8Z7POSZW0485 tag=1RF1VA032PUV type=write size=1611457 2023-12-03T11:30:44.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.053+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=ZH47PG429MDT type=del size=214916: (22) Invalid argument 2023-12-03T11:30:44.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.055+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.0 size=1048576 2023-12-03T11:30:44.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.056+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.1 size=1048576 2023-12-03T11:30:44.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.057+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.2 size=693258 2023-12-03T11:30:44.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.057+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=N0YL0E9WFK0V tag=5K5J88GT6KKJ type=write size=3838986 2023-12-03T11:30:44.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.059+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=86JV74V3WZVZ type=write size=284333 2023-12-03T11:30:44.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.062+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~U5YRE9LL1Z7V.0 size=775871 2023-12-03T11:30:44.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.062+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=U5YRE9LL1Z7V tag=XSWS5JFS7SDY type=write size=1824447 2023-12-03T11:30:44.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.063+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=1OKTM9080G38 type=write size=805429 2023-12-03T11:30:44.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.064+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=XP34M8347J40 type=del size=416746 2023-12-03T11:30:44.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.065+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.0 size=1048576 2023-12-03T11:30:44.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.067+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.1 size=1048576 2023-12-03T11:30:44.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.068+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~HWHDBE4VX6F2.2 size=882327 2023-12-03T11:30:44.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.070+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=HWHDBE4VX6F2 tag=T8YPHUTZN2Z5 type=write size=4028055 2023-12-03T11:30:44.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.072+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~SUNVCVXZEWTX.0 size=1048576 2023-12-03T11:30:44.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.073+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~SUNVCVXZEWTX.1 size=285817 2023-12-03T11:30:44.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.073+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=SUNVCVXZEWTX tag=ETKXC8SDZQYN type=write size=2382969 2023-12-03T11:30:44.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.076+0000 7fc553ded780 1 < suggested update operation key=IRMADF9MXXBZ tag=HX6L1MJ26ZIW type=del size=2380165 2023-12-03T11:30:44.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.077+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=JBA8LCOP28RN type=del size=1269576 2023-12-03T11:30:44.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.078+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=89Q1R3CHK68I type=del 2023-12-03T11:30:44.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.079+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=RCMNU88NFD9C type=del size=3375913 2023-12-03T11:30:44.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.080+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.0 size=1048576 2023-12-03T11:30:44.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.081+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.1 size=1048576 2023-12-03T11:30:44.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.082+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~TVQ16G5Q5L0S.2 size=956927 2023-12-03T11:30:44.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.085+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=TVQ16G5Q5L0S tag=K2XZCICA9RDA type=write size=4102655 2023-12-03T11:30:44.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.086+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=IZRODMN9EOGR type=write size=453331 2023-12-03T11:30:44.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.087+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=7DQ0GF9RT6N4 type=del 2023-12-03T11:30:44.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.088+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=EWRR7OQ1L6GU type=del size=3723455 2023-12-03T11:30:44.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.091+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=PHIVIPWIWTS9 type=del size=3002531 2023-12-03T11:30:44.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.092+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=E2PMS8I4WMET type=del size=3524233 2023-12-03T11:30:44.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.093+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~J26PJJN29MZM.0 size=1048576 2023-12-03T11:30:44.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.094+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~J26PJJN29MZM.1 size=689774 2023-12-03T11:30:44.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.097+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=J26PJJN29MZM tag=UV963H09JDKX type=write size=2786926 2023-12-03T11:30:44.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.098+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=TKDBJHSDGB4K type=del size=512407 2023-12-03T11:30:44.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.099+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~16LZ1BB0JTIO.0 size=1048576 2023-12-03T11:30:44.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.100+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~16LZ1BB0JTIO.1 size=455072 2023-12-03T11:30:44.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.102+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=16LZ1BB0JTIO tag=CA48AJPROFPE type=write size=2552224 2023-12-03T11:30:44.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.104+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=GBIUPIO86K3W type=del size=3492771 2023-12-03T11:30:44.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.105+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.0 size=1048576 2023-12-03T11:30:44.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.106+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.1 size=1048576 2023-12-03T11:30:44.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.107+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~B45YET0BKGCN.2 size=349877 2023-12-03T11:30:44.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.109+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=B45YET0BKGCN tag=NNQWM2JQSGHG type=write size=3495605 2023-12-03T11:30:44.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.111+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~VXTWHLW8LOAN.0 size=1048576 2023-12-03T11:30:44.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.112+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~VXTWHLW8LOAN.1 size=160637 2023-12-03T11:30:44.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.112+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=VXTWHLW8LOAN tag=XPH1XB77TR0M type=write size=2257789 2023-12-03T11:30:44.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.113+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~D2ALGUQ1MYOU.0 size=887284 2023-12-03T11:30:44.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.117+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=D2ALGUQ1MYOU tag=WJ0ZR61HTTDJ type=write size=1935860 2023-12-03T11:30:44.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.119+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~2R5PJK0UFQXN.0 size=805586 2023-12-03T11:30:44.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.119+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=2R5PJK0UFQXN tag=OYM0IWZ8PY5G type=write size=1854162 2023-12-03T11:30:44.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.121+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~3UNAZLK1Z6LT.0 size=1048576 2023-12-03T11:30:44.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.122+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~3UNAZLK1Z6LT.1 size=8600 2023-12-03T11:30:44.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.124+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=3UNAZLK1Z6LT tag=PN6Z4892SC7L type=write size=2105752 2023-12-03T11:30:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.125+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=918FV5LE6IUE type=del size=512066 2023-12-03T11:30:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.126+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=HY1MEZCFNJFP type=del 2023-12-03T11:30:44.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.128+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~AGRIYGM5OUFP.0 size=582276 2023-12-03T11:30:44.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.128+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=AGRIYGM5OUFP tag=LNDYM7A5MJJY type=write size=1630852 2023-12-03T11:30:44.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.130+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~8Z7POSZW0485.0 size=562881 2023-12-03T11:30:44.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.132+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=8Z7POSZW0485 tag=1RF1VA032PUV type=write size=1611457 2023-12-03T11:30:44.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.134+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.0 size=1048576 2023-12-03T11:30:44.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.135+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.1 size=1048576 2023-12-03T11:30:44.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.136+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.2 size=914028 2023-12-03T11:30:44.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.136+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=HE1JG24TX4CL tag=PNQLXRCB7Z6U type=write size=4059756 2023-12-03T11:30:44.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.136+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.0 size=1048576 2023-12-03T11:30:44.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.138+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.1 size=1048576 2023-12-03T11:30:44.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.139+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~N0YL0E9WFK0V.2 size=693258 2023-12-03T11:30:44.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.141+0000 7fc553ded780 1 < canceled multipart upload key=ICDF1EYNI5IU upload=N0YL0E9WFK0V tag=5K5J88GT6KKJ type=write size=3838986 2023-12-03T11:30:44.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.142+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=4EJ87S9K7DG1 type=del size=3194263 2023-12-03T11:30:44.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.144+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~U5YRE9LL1Z7V.0 size=775871 2023-12-03T11:30:44.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.146+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=U5YRE9LL1Z7V tag=XSWS5JFS7SDY type=write size=1824447 2023-12-03T11:30:44.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.147+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=F6X0KD6R4CVO type=del size=3288567 2023-12-03T11:30:44.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.148+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=XP34M8347J40 type=del 2023-12-03T11:30:44.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.151+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~6H2QMSS4LH1F.0 size=1048576 2023-12-03T11:30:44.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.152+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~6H2QMSS4LH1F.1 size=424477 2023-12-03T11:30:44.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.152+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=6H2QMSS4LH1F tag=U5LA3JZCVZF5 type=write size=2521629 2023-12-03T11:30:44.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.153+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~SUNVCVXZEWTX.0 size=1048576 2023-12-03T11:30:44.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.154+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~SUNVCVXZEWTX.1 size=285817 2023-12-03T11:30:44.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.156+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=SUNVCVXZEWTX tag=ETKXC8SDZQYN type=write size=2382969 2023-12-03T11:30:44.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.157+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=1YZ3RSO1L3BV type=del size=2447319 2023-12-03T11:30:44.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.158+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=JBA8LCOP28RN type=del 2023-12-03T11:30:44.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.160+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=8FAIO44R8V9K type=del size=3454868 2023-12-03T11:30:44.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.161+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=RCMNU88NFD9C type=del 2023-12-03T11:30:44.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.162+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=0MCAI3GE6O05 type=del size=215924 2023-12-03T11:30:44.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.164+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=IZRODMN9EOGR type=write size=453331 2023-12-03T11:30:44.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.165+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=K9O28D6O060B type=del size=4032925 2023-12-03T11:30:44.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.167+0000 7fc553ded780 -1 < failed to complete operation key=4TWM1QM1OHOF tag=EWRR7OQ1L6GU type=del size=3723455: (22) Invalid argument 2023-12-03T11:30:44.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.168+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=4BRJUQE60V12 type=del size=2994045 2023-12-03T11:30:44.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.169+0000 7fc553ded780 1 < canceled operation key=ICDF1EYNI5IU tag=E2PMS8I4WMET type=del size=3524233 2023-12-03T11:30:44.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.171+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~7ZY7I301HZ5J.0 size=1000190 2023-12-03T11:30:44.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.171+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=7ZY7I301HZ5J tag=2PESH80L87X2 type=write size=2048766 2023-12-03T11:30:44.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.173+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=TKDBJHSDGB4K type=del 2023-12-03T11:30:44.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.174+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=I5TQAWO9I52Y type=del size=558238 2023-12-03T11:30:44.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.176+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=GBIUPIO86K3W type=del 2023-12-03T11:30:44.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.177+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=IEIDL0DHNPHW type=write size=1004353 2023-12-03T11:30:44.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.179+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~VXTWHLW8LOAN.0 size=1048576 2023-12-03T11:30:44.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.180+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~VXTWHLW8LOAN.1 size=160637 2023-12-03T11:30:44.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.182+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=VXTWHLW8LOAN tag=XPH1XB77TR0M type=write size=2257789 2023-12-03T11:30:44.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.184+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~FMMZEG2AGL9D.0 size=433757 2023-12-03T11:30:44.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.184+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=FMMZEG2AGL9D tag=S7P2XFHIIHXG type=write size=1482333 2023-12-03T11:30:44.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.185+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~2R5PJK0UFQXN.0 size=805586 2023-12-03T11:30:44.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.187+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=2R5PJK0UFQXN tag=OYM0IWZ8PY5G type=write size=1854162 2023-12-03T11:30:44.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.190+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~9UCGML9SC6T2.0 size=1048576 2023-12-03T11:30:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.191+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~9UCGML9SC6T2.1 size=679976 2023-12-03T11:30:44.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.191+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=9UCGML9SC6T2 tag=0N778W9YR13Z type=write size=2777128 2023-12-03T11:30:44.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.192+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=918FV5LE6IUE type=del 2023-12-03T11:30:44.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.193+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=RDFPMQ5TXFAC type=del size=3189350 2023-12-03T11:30:44.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.193+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~AGRIYGM5OUFP.0 size=582276 2023-12-03T11:30:44.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.196+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=AGRIYGM5OUFP tag=LNDYM7A5MJJY type=write size=1630852 2023-12-03T11:30:44.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.197+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=0WHG0EDDC6E3 type=del size=646667 2023-12-03T11:30:44.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.198+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.0 size=1048576 2023-12-03T11:30:44.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.199+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.1 size=1048576 2023-12-03T11:30:44.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.200+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~HE1JG24TX4CL.2 size=914028 2023-12-03T11:30:44.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.203+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=HE1JG24TX4CL tag=PNQLXRCB7Z6U type=write size=4059756 2023-12-03T11:30:44.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.204+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=ZU1BTWZT9VET type=del size=3742924 2023-12-03T11:30:44.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.205+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=4EJ87S9K7DG1 type=del 2023-12-03T11:30:44.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.207+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=8W7IQVO9A9PF type=del size=1327008 2023-12-03T11:30:44.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.208+0000 7fc553ded780 1 < canceled operation key=ICDF1EYNI5IU tag=F6X0KD6R4CVO type=del size=3288567 2023-12-03T11:30:44.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.210+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~XU2IGKRJPHZF.0 size=473533 2023-12-03T11:30:44.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.210+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=XU2IGKRJPHZF tag=FGLHRAJYBTF3 type=write size=1522109 2023-12-03T11:30:44.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.211+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~6H2QMSS4LH1F.0 size=1048576 2023-12-03T11:30:44.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.212+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~6H2QMSS4LH1F.1 size=424477 2023-12-03T11:30:44.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.214+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=6H2QMSS4LH1F tag=U5LA3JZCVZF5 type=write size=2521629 2023-12-03T11:30:44.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.217+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~3617U9J0OQUT.0 size=1048576 2023-12-03T11:30:44.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.218+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~3617U9J0OQUT.1 size=544984 2023-12-03T11:30:44.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.218+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=3617U9J0OQUT tag=9LWBMJ082079 type=write size=2642136 2023-12-03T11:30:44.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.219+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=1YZ3RSO1L3BV type=del 2023-12-03T11:30:44.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.220+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=U7THE4REYF70 type=del size=356102 2023-12-03T11:30:44.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.223+0000 7fc553ded780 1 < suggested update operation key=IRMADF9MXXBZ tag=8FAIO44R8V9K type=del size=3454868 2023-12-03T11:30:44.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.224+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=K9CKPGUO44ZZ type=write size=198001 2023-12-03T11:30:44.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.225+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=0MCAI3GE6O05 type=del 2023-12-03T11:30:44.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.226+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=CEAFTHKOMW6D type=del size=963601 2023-12-03T11:30:44.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.228+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=K9O28D6O060B type=del 2023-12-03T11:30:44.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.229+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=UXRNBE19OI07 type=del size=3241671 2023-12-03T11:30:44.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.230+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=4BRJUQE60V12 type=del 2023-12-03T11:30:44.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.233+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HX7IB2QA3RCN.0 size=1048576 2023-12-03T11:30:44.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.234+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HX7IB2QA3RCN.1 size=895738 2023-12-03T11:30:44.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.234+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=HX7IB2QA3RCN tag=THKRSP0NIAGE type=write size=2992890 2023-12-03T11:30:44.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.235+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~7ZY7I301HZ5J.0 size=1000190 2023-12-03T11:30:44.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.237+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=7ZY7I301HZ5J tag=2PESH80L87X2 type=write size=2048766 2023-12-03T11:30:44.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.238+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=KRKTYA9UA3MR type=del size=1547891 2023-12-03T11:30:44.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.239+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=I5TQAWO9I52Y type=del 2023-12-03T11:30:44.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.240+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=VFD3L5EGYIB5 type=del size=2642316 2023-12-03T11:30:44.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.242+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=IEIDL0DHNPHW type=write size=1004353 2023-12-03T11:30:44.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.243+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=11MZ02ET5Q6L type=write size=415566 2023-12-03T11:30:44.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.244+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~FMMZEG2AGL9D.0 size=433757 2023-12-03T11:30:44.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.248+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=FMMZEG2AGL9D tag=S7P2XFHIIHXG type=write size=1482333 2023-12-03T11:30:44.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.252+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~74D41GVX5XS0.0 size=978514 2023-12-03T11:30:44.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.252+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=74D41GVX5XS0 tag=389II2PQL8SH type=write size=2027090 2023-12-03T11:30:44.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.253+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~9UCGML9SC6T2.0 size=1048576 2023-12-03T11:30:44.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.254+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~9UCGML9SC6T2.1 size=679976 2023-12-03T11:30:44.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.256+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=9UCGML9SC6T2 tag=0N778W9YR13Z type=write size=2777128 2023-12-03T11:30:44.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.258+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=05AWMRU4UX3Y type=del size=1485090 2023-12-03T11:30:44.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.262+0000 7fc553ded780 1 < suggested remove operation key=588S2ASN9I0W tag=RDFPMQ5TXFAC type=del 2023-12-03T11:30:44.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.263+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=9I2Q5O7G5JYR type=write size=587016 2023-12-03T11:30:44.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.265+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=0WHG0EDDC6E3 type=del 2023-12-03T11:30:44.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.267+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.0 size=1048576 2023-12-03T11:30:44.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.268+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.1 size=1048576 2023-12-03T11:30:44.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.269+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.2 size=843858 2023-12-03T11:30:44.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.269+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=HCX7MZTDF7RK tag=GQLXCMME4L6A type=write size=3989586 2023-12-03T11:30:44.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.271+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=ZU1BTWZT9VET type=del 2023-12-03T11:30:44.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.273+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~X0F7EBDYDUA0.0 size=326373 2023-12-03T11:30:44.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.273+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=X0F7EBDYDUA0 tag=1A85RDRNYG40 type=write size=1374949 2023-12-03T11:30:44.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.274+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=8W7IQVO9A9PF type=del 2023-12-03T11:30:44.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.277+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~3EZE7GUL4UTC.0 size=1048576 2023-12-03T11:30:44.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.278+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~3EZE7GUL4UTC.1 size=245145 2023-12-03T11:30:44.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.278+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=3EZE7GUL4UTC tag=95GR9SAUF2L7 type=write size=2342297 2023-12-03T11:30:44.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.279+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~XU2IGKRJPHZF.0 size=473533 2023-12-03T11:30:44.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.282+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=XU2IGKRJPHZF tag=FGLHRAJYBTF3 type=write size=1522109 2023-12-03T11:30:44.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.283+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=RJ3P5LYWAFBW type=del size=1516403 2023-12-03T11:30:44.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.284+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~3617U9J0OQUT.0 size=1048576 2023-12-03T11:30:44.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.285+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~3617U9J0OQUT.1 size=544984 2023-12-03T11:30:44.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.288+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=3617U9J0OQUT tag=9LWBMJ082079 type=write size=2642136 2023-12-03T11:30:44.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.289+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=D1FI9K0NOWQM type=del size=144317 2023-12-03T11:30:44.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.292+0000 7fc553ded780 1 < suggested update operation key=706CET5PSDEG tag=U7THE4REYF70 type=del size=356102 2023-12-03T11:30:44.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.294+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.0 size=1048576 2023-12-03T11:30:44.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.295+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.1 size=1048576 2023-12-03T11:30:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.296+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.2 size=411755 2023-12-03T11:30:44.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.296+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=GG77GFNAUKUS tag=7MDXCZU0VS3Q type=write size=3557483 2023-12-03T11:30:44.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.298+0000 7fc553ded780 1 < completed write operation key=6OW429S51UWN tag=K9CKPGUO44ZZ type=write size=198001 2023-12-03T11:30:44.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.299+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=VMSKJ7UJGQTW type=write size=697491 2023-12-03T11:30:44.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.300+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=CEAFTHKOMW6D type=del 2023-12-03T11:30:44.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.303+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.0 size=1048576 2023-12-03T11:30:44.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.304+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.1 size=1048576 2023-12-03T11:30:44.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.305+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.2 size=191292 2023-12-03T11:30:44.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.305+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=GYVK0QAZSL23 tag=8861QXUNEPQ7 type=write size=3337020 2023-12-03T11:30:44.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.306+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=UXRNBE19OI07 type=del 2023-12-03T11:30:44.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.309+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~5TP00JBK5D3L.0 size=751256 2023-12-03T11:30:44.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.309+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=5TP00JBK5D3L tag=NDSK9HXYMQWJ type=write size=1799832 2023-12-03T11:30:44.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.309+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~HX7IB2QA3RCN.0 size=1048576 2023-12-03T11:30:44.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.310+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HX7IB2QA3RCN.1 size=895738 2023-12-03T11:30:44.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.312+0000 7fc553ded780 1 < canceled multipart upload key=ICDF1EYNI5IU upload=HX7IB2QA3RCN tag=THKRSP0NIAGE type=write size=2992890 2023-12-03T11:30:44.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.314+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=4KPCH70MF4LC type=del size=373167 2023-12-03T11:30:44.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.315+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=KRKTYA9UA3MR type=del 2023-12-03T11:30:44.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.316+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=DH3D2MMQT571 type=del size=833873 2023-12-03T11:30:44.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.318+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=VFD3L5EGYIB5 type=del 2023-12-03T11:30:44.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.319+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=K5RD8AGUJ38M type=write size=581620 2023-12-03T11:30:44.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.321+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=11MZ02ET5Q6L type=write size=415566 2023-12-03T11:30:44.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.322+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=G0H25AF4956D type=del size=2944081 2023-12-03T11:30:44.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.323+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~74D41GVX5XS0.0 size=978514 2023-12-03T11:30:44.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.325+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=74D41GVX5XS0 tag=389II2PQL8SH type=write size=2027090 2023-12-03T11:30:44.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.326+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=L19LIAWFNQQT type=del size=1253614 2023-12-03T11:30:44.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.327+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=05AWMRU4UX3Y type=del 2023-12-03T11:30:44.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.330+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~I12WG0O6JBOD.0 size=882571 2023-12-03T11:30:44.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.330+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=I12WG0O6JBOD tag=6DLS672S4MQX type=write size=1931147 2023-12-03T11:30:44.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.332+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=9I2Q5O7G5JYR type=write size=587016 2023-12-03T11:30:44.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.333+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=MJFYOLRCH43Q type=del size=1508201 2023-12-03T11:30:44.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.334+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.0 size=1048576 2023-12-03T11:30:44.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.335+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.1 size=1048576 2023-12-03T11:30:44.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.336+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~HCX7MZTDF7RK.2 size=843858 2023-12-03T11:30:44.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.338+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=HCX7MZTDF7RK tag=GQLXCMME4L6A type=write size=3989586 2023-12-03T11:30:44.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.340+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~TL6WL03KHBV4.0 size=1048576 2023-12-03T11:30:44.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.341+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~TL6WL03KHBV4.1 size=873140 2023-12-03T11:30:44.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.341+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=TL6WL03KHBV4 tag=4H5041B926ES type=write size=2970292 2023-12-03T11:30:44.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.342+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~X0F7EBDYDUA0.0 size=326373 2023-12-03T11:30:44.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.344+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=X0F7EBDYDUA0 tag=1A85RDRNYG40 type=write size=1374949 2023-12-03T11:30:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.345+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=RC3CAHS5F9T3 type=del size=2180650 2023-12-03T11:30:44.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.345+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~3EZE7GUL4UTC.0 size=1048576 2023-12-03T11:30:44.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.346+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~3EZE7GUL4UTC.1 size=245145 2023-12-03T11:30:44.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.348+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=3EZE7GUL4UTC tag=95GR9SAUF2L7 type=write size=2342297 2023-12-03T11:30:44.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.349+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=U7GNIU8UHV16 type=del size=3033219 2023-12-03T11:30:44.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.351+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=RJ3P5LYWAFBW type=del 2023-12-03T11:30:44.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.353+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~4A3Y9QXJM7BQ.0 size=1048576 2023-12-03T11:30:44.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.353+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~4A3Y9QXJM7BQ.1 size=1002549 2023-12-03T11:30:44.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.353+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=4A3Y9QXJM7BQ tag=Y85208M0M1MQ type=write size=3099701 2023-12-03T11:30:44.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.355+0000 7fc553ded780 -1 < failed to complete operation key=706CET5PSDEG tag=D1FI9K0NOWQM type=del size=144317: (22) Invalid argument 2023-12-03T11:30:44.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.356+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=MX1OO4CFNUAY type=del size=3408545 2023-12-03T11:30:44.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.357+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.0 size=1048576 2023-12-03T11:30:44.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.358+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.1 size=1048576 2023-12-03T11:30:44.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.359+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~GG77GFNAUKUS.2 size=411755 2023-12-03T11:30:44.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.361+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=GG77GFNAUKUS tag=7MDXCZU0VS3Q type=write size=3557483 2023-12-03T11:30:44.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.363+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.0 size=1048576 2023-12-03T11:30:44.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.364+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.1 size=1048576 2023-12-03T11:30:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.365+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.2 size=497998 2023-12-03T11:30:44.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.365+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=V381Q9NET19Y tag=44DF7KLTVHLJ type=write size=3643726 2023-12-03T11:30:44.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.367+0000 7fc553ded780 1 < suggested remove operation key=0H5OZMBMLQFJ tag=VMSKJ7UJGQTW type=write 2023-12-03T11:30:44.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.368+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=8PXAHYZHO11A type=del size=1489047 2023-12-03T11:30:44.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.370+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.0 size=1048576 2023-12-03T11:30:44.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.371+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.1 size=1048576 2023-12-03T11:30:44.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.371+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~GYVK0QAZSL23.2 size=191292 2023-12-03T11:30:44.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.374+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=GYVK0QAZSL23 tag=8861QXUNEPQ7 type=write size=3337020 2023-12-03T11:30:44.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.375+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=6899E7RPXCZY type=del size=1761924 2023-12-03T11:30:44.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.376+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~5TP00JBK5D3L.0 size=751256 2023-12-03T11:30:44.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.379+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=5TP00JBK5D3L tag=NDSK9HXYMQWJ type=write size=1799832 2023-12-03T11:30:44.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.380+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=8T7LGN87IBR4 type=del size=1031137 2023-12-03T11:30:44.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.381+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=4KPCH70MF4LC type=del 2023-12-03T11:30:44.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.389+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=SQ83INR7C5MA type=del size=2148560 2023-12-03T11:30:44.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.390+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=DH3D2MMQT571 type=del 2023-12-03T11:30:44.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.391+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=BRBUXNBFY4IG type=del size=3537510 2023-12-03T11:30:44.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.393+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=K5RD8AGUJ38M type=write size=581620 2023-12-03T11:30:44.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.395+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.0 size=1048576 2023-12-03T11:30:44.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.396+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.1 size=1048576 2023-12-03T11:30:44.398 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.397+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.2 size=846541 2023-12-03T11:30:44.398 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.397+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=U91A981E4TVU tag=5XGA5H154IC4 type=write size=3992269 2023-12-03T11:30:44.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.399+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=G0H25AF4956D type=del size=2944081 2023-12-03T11:30:44.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.401+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PX50K0TP9CW0.0 size=1048576 2023-12-03T11:30:44.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.402+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PX50K0TP9CW0.1 size=696293 2023-12-03T11:30:44.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.402+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=PX50K0TP9CW0 tag=8U9BBF7W3GW4 type=write size=2793445 2023-12-03T11:30:44.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.404+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=L19LIAWFNQQT type=del 2023-12-03T11:30:44.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.405+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=R0I1SMSF3YXG type=del size=3357477 2023-12-03T11:30:44.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.406+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~I12WG0O6JBOD.0 size=882571 2023-12-03T11:30:44.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.408+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=I12WG0O6JBOD tag=6DLS672S4MQX type=write size=1931147 2023-12-03T11:30:44.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.410+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~5M6E1M4A0TIQ.0 size=1048576 2023-12-03T11:30:44.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.411+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~5M6E1M4A0TIQ.1 size=38462 2023-12-03T11:30:44.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.411+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=5M6E1M4A0TIQ tag=NQ1DWC6NWLDK type=write size=2135614 2023-12-03T11:30:44.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.413+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=MJFYOLRCH43Q type=del 2023-12-03T11:30:44.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.415+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~BIHO5307STND.0 size=533317 2023-12-03T11:30:44.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.415+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=BIHO5307STND tag=QUP47ZD13K68 type=write size=1581893 2023-12-03T11:30:44.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.416+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~TL6WL03KHBV4.0 size=1048576 2023-12-03T11:30:44.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.417+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~TL6WL03KHBV4.1 size=873140 2023-12-03T11:30:44.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.419+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=TL6WL03KHBV4 tag=4H5041B926ES type=write size=2970292 2023-12-03T11:30:44.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.420+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=ITTVRS4HN1ZJ type=del size=360950 2023-12-03T11:30:44.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.421+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=RC3CAHS5F9T3 type=del 2023-12-03T11:30:44.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.422+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=GR6QAAQOS1N7 type=del size=842477 2023-12-03T11:30:44.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.425+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=U7GNIU8UHV16 type=del 2023-12-03T11:30:44.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.426+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=AMEYBM1UYLU8 type=del size=1010502 2023-12-03T11:30:44.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.427+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~4A3Y9QXJM7BQ.0 size=1048576 2023-12-03T11:30:44.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.428+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~4A3Y9QXJM7BQ.1 size=1002549 2023-12-03T11:30:44.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.430+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=4A3Y9QXJM7BQ tag=Y85208M0M1MQ type=write size=3099701 2023-12-03T11:30:44.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.431+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=KW9EF2Q13MMS type=del size=1608739 2023-12-03T11:30:44.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.432+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=MX1OO4CFNUAY type=del 2023-12-03T11:30:44.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.434+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.0 size=1048576 2023-12-03T11:30:44.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.435+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.1 size=1048576 2023-12-03T11:30:44.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.436+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.2 size=273847 2023-12-03T11:30:44.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.436+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=MRWXJVSOVS5X tag=XVO783WIQ8F6 type=write size=3419575 2023-12-03T11:30:44.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.437+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.0 size=1048576 2023-12-03T11:30:44.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.438+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.1 size=1048576 2023-12-03T11:30:44.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.439+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~V381Q9NET19Y.2 size=497998 2023-12-03T11:30:44.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.441+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=V381Q9NET19Y tag=44DF7KLTVHLJ type=write size=3643726 2023-12-03T11:30:44.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.443+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.0 size=1048576 2023-12-03T11:30:44.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.444+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.1 size=1048576 2023-12-03T11:30:44.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.445+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.2 size=1019126 2023-12-03T11:30:44.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.445+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=ZUZYZ3FX95DB tag=4WCH9N0IAYHZ type=write size=4164854 2023-12-03T11:30:44.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.446+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=8PXAHYZHO11A type=del 2023-12-03T11:30:44.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.449+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~01A026233J2L.0 size=996860 2023-12-03T11:30:44.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.449+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=01A026233J2L tag=C2PQ0WM10CY2 type=write size=2045436 2023-12-03T11:30:44.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.451+0000 7fc553ded780 1 < suggested remove operation key=ZUNOFY5N3J3H tag=6899E7RPXCZY type=del 2023-12-03T11:30:44.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.452+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=HAZSV6TJ3N7G type=write size=920409 2023-12-03T11:30:44.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.453+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=8T7LGN87IBR4 type=del 2023-12-03T11:30:44.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.455+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=S5MTICRQV0QR type=del size=104255 2023-12-03T11:30:44.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.456+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=SQ83INR7C5MA type=del 2023-12-03T11:30:44.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.457+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=SZNT0PZG9VCP type=write size=1011691 2023-12-03T11:30:44.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.458+0000 7fc553ded780 1 < canceled operation key=SWTRHXTAR0A1 tag=BRBUXNBFY4IG type=del size=3537510 2023-12-03T11:30:44.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.461+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~JHP2FBOQ6U5D.0 size=845576 2023-12-03T11:30:44.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.461+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=JHP2FBOQ6U5D tag=2JWJ0DVQRB0F type=write size=1894152 2023-12-03T11:30:44.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.462+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.0 size=1048576 2023-12-03T11:30:44.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.463+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.1 size=1048576 2023-12-03T11:30:44.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.464+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~U91A981E4TVU.2 size=846541 2023-12-03T11:30:44.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.466+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=U91A981E4TVU tag=5XGA5H154IC4 type=write size=3992269 2023-12-03T11:30:44.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.468+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~DNCR2LZ1I2RH.0 size=77394 2023-12-03T11:30:44.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.468+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=DNCR2LZ1I2RH tag=PF7DVCGY1NZQ type=write size=1125970 2023-12-03T11:30:44.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.468+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~PX50K0TP9CW0.0 size=1048576 2023-12-03T11:30:44.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.469+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~PX50K0TP9CW0.1 size=696293 2023-12-03T11:30:44.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.472+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=PX50K0TP9CW0 tag=8U9BBF7W3GW4 type=write size=2793445 2023-12-03T11:30:44.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.473+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=A15MIJKUKV73 type=del size=1786513 2023-12-03T11:30:44.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.474+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=R0I1SMSF3YXG type=del 2023-12-03T11:30:44.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.476+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=TTTPD977M8KR type=del size=2947918 2023-12-03T11:30:44.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.477+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~5M6E1M4A0TIQ.0 size=1048576 2023-12-03T11:30:44.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.478+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~5M6E1M4A0TIQ.1 size=38462 2023-12-03T11:30:44.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.480+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=5M6E1M4A0TIQ tag=NQ1DWC6NWLDK type=write size=2135614 2023-12-03T11:30:44.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.482+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~26IAJ1PAD1HW.0 size=189777 2023-12-03T11:30:44.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.482+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=26IAJ1PAD1HW tag=VOX34P5Y7SGU type=write size=1238353 2023-12-03T11:30:44.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.483+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~BIHO5307STND.0 size=533317 2023-12-03T11:30:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.485+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=BIHO5307STND tag=QUP47ZD13K68 type=write size=1581893 2023-12-03T11:30:44.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.487+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~QW9F6JTOUMJJ.0 size=1048576 2023-12-03T11:30:44.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.488+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~QW9F6JTOUMJJ.1 size=594474 2023-12-03T11:30:44.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.488+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=QW9F6JTOUMJJ tag=RS3OUVFXONYB type=write size=2691626 2023-12-03T11:30:44.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.489+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=ITTVRS4HN1ZJ type=del 2023-12-03T11:30:44.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.491+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=UKVCJU9A9JZB type=del size=1520124 2023-12-03T11:30:44.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.492+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=GR6QAAQOS1N7 type=del 2023-12-03T11:30:44.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.493+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=ZQP0C17EMWW1 type=write size=403563 2023-12-03T11:30:44.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.494+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=AMEYBM1UYLU8 type=del 2023-12-03T11:30:44.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.498+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~L6OZIPOVPL3V.0 size=514994 2023-12-03T11:30:44.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.498+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=L6OZIPOVPL3V tag=SGU6MCCP6K2I type=write size=1563570 2023-12-03T11:30:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.501+0000 7fc553ded780 1 < suggested update operation key=PL22GPUFN3O0 tag=KW9EF2Q13MMS type=del size=1608739 2023-12-03T11:30:44.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.502+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=2R0KA0RPS32A type=write size=269583 2023-12-03T11:30:44.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.503+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.0 size=1048576 2023-12-03T11:30:44.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.504+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.1 size=1048576 2023-12-03T11:30:44.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.505+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~MRWXJVSOVS5X.2 size=273847 2023-12-03T11:30:44.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.507+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=MRWXJVSOVS5X tag=XVO783WIQ8F6 type=write size=3419575 2023-12-03T11:30:44.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.508+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=Z0ABU1540S7X type=del size=1214345 2023-12-03T11:30:44.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.510+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.0 size=1048576 2023-12-03T11:30:44.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.511+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.1 size=1048576 2023-12-03T11:30:44.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.511+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~ZUZYZ3FX95DB.2 size=1019126 2023-12-03T11:30:44.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.513+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=ZUZYZ3FX95DB tag=4WCH9N0IAYHZ type=write size=4164854 2023-12-03T11:30:44.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.514+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=MN93XUENYXPN type=del size=2621366 2023-12-03T11:30:44.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.515+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~01A026233J2L.0 size=996860 2023-12-03T11:30:44.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.517+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=01A026233J2L tag=C2PQ0WM10CY2 type=write size=2045436 2023-12-03T11:30:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.518+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=RF4C92H08985 type=del size=1954954 2023-12-03T11:30:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.519+0000 7fc553ded780 1 < completed write operation key=706CET5PSDEG tag=HAZSV6TJ3N7G type=write size=920409 2023-12-03T11:30:44.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.522+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~0S4YFC351M4W.0 size=1048576 2023-12-03T11:30:44.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.523+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~0S4YFC351M4W.1 size=858307 2023-12-03T11:30:44.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.523+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=0S4YFC351M4W tag=9WR3FVV7EJEY type=write size=2955459 2023-12-03T11:30:44.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.524+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=S5MTICRQV0QR type=del 2023-12-03T11:30:44.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.526+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.0 size=1048576 2023-12-03T11:30:44.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.527+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.1 size=1048576 2023-12-03T11:30:44.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.528+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.2 size=267397 2023-12-03T11:30:44.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.528+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=WAJLTLB52NV6 tag=QCXZA45HEA7A type=write size=3413125 2023-12-03T11:30:44.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.529+0000 7fc553ded780 1 < canceled operation key=FC4WU6SMNEWV tag=SZNT0PZG9VCP type=write size=1011691 2023-12-03T11:30:44.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.531+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=NYPY3XXBPBEX type=del size=3215174 2023-12-03T11:30:44.533 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.532+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~JHP2FBOQ6U5D.0 size=845576 2023-12-03T11:30:44.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.534+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=JHP2FBOQ6U5D tag=2JWJ0DVQRB0F type=write size=1894152 2023-12-03T11:30:44.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.536+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.0 size=1048576 2023-12-03T11:30:44.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.537+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.1 size=1048576 2023-12-03T11:30:44.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.538+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.2 size=997166 2023-12-03T11:30:44.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.538+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=6F0E2P0NO05C tag=RZ5YAVSZHLAS type=write size=4142894 2023-12-03T11:30:44.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.539+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~DNCR2LZ1I2RH.0 size=77394 2023-12-03T11:30:44.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.542+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=DNCR2LZ1I2RH tag=PF7DVCGY1NZQ type=write size=1125970 2023-12-03T11:30:44.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.544+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.0 size=1048576 2023-12-03T11:30:44.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.545+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.1 size=1048576 2023-12-03T11:30:44.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.546+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.2 size=493391 2023-12-03T11:30:44.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.546+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=BY409AK7QJGQ tag=GKMRBGKR1N45 type=write size=3639119 2023-12-03T11:30:44.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.547+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=A15MIJKUKV73 type=del 2023-12-03T11:30:44.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.549+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~S7YCY80A8RVJ.0 size=1048576 2023-12-03T11:30:44.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.551+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~S7YCY80A8RVJ.1 size=63507 2023-12-03T11:30:44.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.551+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=S7YCY80A8RVJ tag=HENH8A98GJTF type=write size=2160659 2023-12-03T11:30:44.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.552+0000 7fc553ded780 1 < canceled operation key=L4ME9RYRR4NG tag=TTTPD977M8KR type=del size=2947918 2023-12-03T11:30:44.554 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.553+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=MJQ1YBDZWVS1 type=del size=1190625 2023-12-03T11:30:44.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.555+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~26IAJ1PAD1HW.0 size=189777 2023-12-03T11:30:44.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.557+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=26IAJ1PAD1HW tag=VOX34P5Y7SGU type=write size=1238353 2023-12-03T11:30:44.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.559+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=XYGC2W7JYAAR type=del size=2689713 2023-12-03T11:30:44.561 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.560+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~QW9F6JTOUMJJ.0 size=1048576 2023-12-03T11:30:44.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.561+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~QW9F6JTOUMJJ.1 size=594474 2023-12-03T11:30:44.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.563+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=QW9F6JTOUMJJ tag=RS3OUVFXONYB type=write size=2691626 2023-12-03T11:30:44.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.564+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=BT2K7WB766Z4 type=del size=166191 2023-12-03T11:30:44.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.565+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=UKVCJU9A9JZB type=del 2023-12-03T11:30:44.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.566+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=7O1YMSBLG3AX type=write size=268824 2023-12-03T11:30:44.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.568+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=ZQP0C17EMWW1 type=write size=403563 2023-12-03T11:30:44.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.570+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.0 size=1048576 2023-12-03T11:30:44.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.571+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.1 size=1048576 2023-12-03T11:30:44.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.572+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.2 size=1029885 2023-12-03T11:30:44.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.572+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=4Z4Z781KBJ9G tag=MNAQ06IDR639 type=write size=4175613 2023-12-03T11:30:44.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.573+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~L6OZIPOVPL3V.0 size=514994 2023-12-03T11:30:44.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.575+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=L6OZIPOVPL3V tag=SGU6MCCP6K2I type=write size=1563570 2023-12-03T11:30:44.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.576+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=DE8ORFT6I104 type=del size=1272192 2023-12-03T11:30:44.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.579+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=2R0KA0RPS32A type=write size=269583 2023-12-03T11:30:44.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.581+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.0 size=1048576 2023-12-03T11:30:44.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.583+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.1 size=1048576 2023-12-03T11:30:44.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.584+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.2 size=71082 2023-12-03T11:30:44.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.584+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=7SE086Q6AY23 tag=XMOZCR2H6HHH type=write size=3216810 2023-12-03T11:30:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.585+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=Z0ABU1540S7X type=del 2023-12-03T11:30:44.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.586+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=EQ0Z7L0SU4NA type=del size=2716062 2023-12-03T11:30:44.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.589+0000 7fc553ded780 1 < suggested remove operation key=X741JIRT7RIG tag=MN93XUENYXPN type=del 2023-12-03T11:30:44.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.593+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~1FJ5E0LSPY8P.0 size=919358 2023-12-03T11:30:44.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.593+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=1FJ5E0LSPY8P tag=URW74RJB9MK2 type=write size=1967934 2023-12-03T11:30:44.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.594+0000 7fc553ded780 -1 < failed to complete operation key=X741JIRT7RIG tag=RF4C92H08985 type=del size=1954954: (22) Invalid argument 2023-12-03T11:30:44.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.595+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=RBN4GA9CGMOB type=del size=2485923 2023-12-03T11:30:44.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.596+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~0S4YFC351M4W.0 size=1048576 2023-12-03T11:30:44.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.597+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~0S4YFC351M4W.1 size=858307 2023-12-03T11:30:44.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.600+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=0S4YFC351M4W tag=9WR3FVV7EJEY type=write size=2955459 2023-12-03T11:30:44.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.602+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~25ZXL9JDEPI7.0 size=1048576 2023-12-03T11:30:44.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.603+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~25ZXL9JDEPI7.1 size=506190 2023-12-03T11:30:44.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.603+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=25ZXL9JDEPI7 tag=DDXGUH9SN8R0 type=write size=2603342 2023-12-03T11:30:44.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.604+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.0 size=1048576 2023-12-03T11:30:44.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.605+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.1 size=1048576 2023-12-03T11:30:44.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.606+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~WAJLTLB52NV6.2 size=267397 2023-12-03T11:30:44.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.608+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=WAJLTLB52NV6 tag=QCXZA45HEA7A type=write size=3413125 2023-12-03T11:30:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.611+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~Y9RF3N1JZGQS.0 size=1048576 2023-12-03T11:30:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.612+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~Y9RF3N1JZGQS.1 size=864905 2023-12-03T11:30:44.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.612+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=Y9RF3N1JZGQS tag=CKYM24XWA5DX type=write size=2962057 2023-12-03T11:30:44.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.613+0000 7fc553ded780 1 < canceled operation key=4TWM1QM1OHOF tag=NYPY3XXBPBEX type=del size=3215174 2023-12-03T11:30:44.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.614+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=GRMFHO0OP074 type=del size=2287013 2023-12-03T11:30:44.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.616+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.0 size=1048576 2023-12-03T11:30:44.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.617+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.1 size=1048576 2023-12-03T11:30:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.618+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~6F0E2P0NO05C.2 size=997166 2023-12-03T11:30:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.621+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=6F0E2P0NO05C tag=RZ5YAVSZHLAS type=write size=4142894 2023-12-03T11:30:44.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.622+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=812J0IHCIH21 type=write size=204859 2023-12-03T11:30:44.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.623+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.0 size=1048576 2023-12-03T11:30:44.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.623+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.1 size=1048576 2023-12-03T11:30:44.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.624+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~BY409AK7QJGQ.2 size=493391 2023-12-03T11:30:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.626+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=BY409AK7QJGQ tag=GKMRBGKR1N45 type=write size=3639119 2023-12-03T11:30:44.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.627+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=9MJ7GKG3SXD5 type=del size=2094032 2023-12-03T11:30:44.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.629+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~S7YCY80A8RVJ.0 size=1048576 2023-12-03T11:30:44.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.630+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~S7YCY80A8RVJ.1 size=63507 2023-12-03T11:30:44.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.632+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=S7YCY80A8RVJ tag=HENH8A98GJTF type=write size=2160659 2023-12-03T11:30:44.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.633+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=ZXUAT30F8KWA type=write size=697701 2023-12-03T11:30:44.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.635+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=MJQ1YBDZWVS1 type=del 2023-12-03T11:30:44.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.637+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~UK8IJW1CN5HF.0 size=238863 2023-12-03T11:30:44.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.637+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=UK8IJW1CN5HF tag=3RDQ22XH83W2 type=write size=1287439 2023-12-03T11:30:44.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.638+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=XYGC2W7JYAAR type=del 2023-12-03T11:30:44.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.640+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=2PLAOXGSO5I1 type=del size=3239001 2023-12-03T11:30:44.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.641+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=BT2K7WB766Z4 type=del 2023-12-03T11:30:44.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.642+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=FXF7OZTR2C8P type=del size=1542864 2023-12-03T11:30:44.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.643+0000 7fc553ded780 1 < completed write operation key=J664UUE697Q8 tag=7O1YMSBLG3AX type=write size=268824 2023-12-03T11:30:44.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.645+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=D4SX2EJP6Q7G type=write size=492144 2023-12-03T11:30:44.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.646+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.0 size=1048576 2023-12-03T11:30:44.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.647+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.1 size=1048576 2023-12-03T11:30:44.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.648+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~4Z4Z781KBJ9G.2 size=1029885 2023-12-03T11:30:44.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.650+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=4Z4Z781KBJ9G tag=MNAQ06IDR639 type=write size=4175613 2023-12-03T11:30:44.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.651+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=9RZKLABB62TV type=del size=1601958 2023-12-03T11:30:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.653+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=DE8ORFT6I104 type=del 2023-12-03T11:30:44.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.655+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PDUTW0DM9SRA.0 size=154410 2023-12-03T11:30:44.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.655+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=PDUTW0DM9SRA tag=SCW41ULTFD61 type=write size=1202986 2023-12-03T11:30:44.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.655+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.0 size=1048576 2023-12-03T11:30:44.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.656+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.1 size=1048576 2023-12-03T11:30:44.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.657+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~7SE086Q6AY23.2 size=71082 2023-12-03T11:30:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.659+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=7SE086Q6AY23 tag=XMOZCR2H6HHH type=write size=3216810 2023-12-03T11:30:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.660+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=ONOLFY2T9TAU type=del size=2541600 2023-12-03T11:30:44.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.661+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=EQ0Z7L0SU4NA type=del 2023-12-03T11:30:44.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.664+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~24OX4574JV1Q.0 size=837223 2023-12-03T11:30:44.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.664+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=24OX4574JV1Q tag=XJDADRZUBQY4 type=write size=1885799 2023-12-03T11:30:44.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.665+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~1FJ5E0LSPY8P.0 size=919358 2023-12-03T11:30:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.667+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=1FJ5E0LSPY8P tag=URW74RJB9MK2 type=write size=1967934 2023-12-03T11:30:44.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.669+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~KZ2Q7S9CN5MH.0 size=190304 2023-12-03T11:30:44.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.669+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=KZ2Q7S9CN5MH tag=WET240KHXDAS type=write size=1238880 2023-12-03T11:30:44.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.671+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=RBN4GA9CGMOB type=del 2023-12-03T11:30:44.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.672+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=YO3SXS0KQUDM type=write size=474617 2023-12-03T11:30:44.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.673+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~25ZXL9JDEPI7.0 size=1048576 2023-12-03T11:30:44.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.674+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~25ZXL9JDEPI7.1 size=506190 2023-12-03T11:30:44.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.676+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=25ZXL9JDEPI7 tag=DDXGUH9SN8R0 type=write size=2603342 2023-12-03T11:30:44.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.678+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~WEXV56T483WT.0 size=66321 2023-12-03T11:30:44.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.678+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=WEXV56T483WT tag=VIRT12581IDY type=write size=1114897 2023-12-03T11:30:44.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.679+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~Y9RF3N1JZGQS.0 size=1048576 2023-12-03T11:30:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.680+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~Y9RF3N1JZGQS.1 size=864905 2023-12-03T11:30:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.683+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=Y9RF3N1JZGQS tag=CKYM24XWA5DX type=write size=2962057 2023-12-03T11:30:44.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.685+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~NFX7W842I7MG.0 size=925991 2023-12-03T11:30:44.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.685+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=NFX7W842I7MG tag=497JZJVS16U8 type=write size=1974567 2023-12-03T11:30:44.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.686+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=GRMFHO0OP074 type=del 2023-12-03T11:30:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.689+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.0 size=1048576 2023-12-03T11:30:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.690+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.1 size=1048576 2023-12-03T11:30:44.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.691+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.2 size=243131 2023-12-03T11:30:44.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.691+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=I57T50Z1NYOQ tag=7WMK20UH09RP type=write size=3388859 2023-12-03T11:30:44.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.692+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=812J0IHCIH21 type=write size=204859 2023-12-03T11:30:44.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.694+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=BEML16ZVO5PS type=del size=1146169 2023-12-03T11:30:44.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.695+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=9MJ7GKG3SXD5 type=del 2023-12-03T11:30:44.697 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.696+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=BAK3CU0OODN1 type=del size=2021168 2023-12-03T11:30:44.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.699+0000 7fc553ded780 1 < suggested remove operation key=C2TPF9Q5Z7XQ tag=ZXUAT30F8KWA type=write 2023-12-03T11:30:44.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.701+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~3XPIUZBTB8NT.0 size=1048576 2023-12-03T11:30:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.702+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~3XPIUZBTB8NT.1 size=178118 2023-12-03T11:30:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.702+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=3XPIUZBTB8NT tag=BU0C46PNT8SX type=write size=2275270 2023-12-03T11:30:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.703+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~UK8IJW1CN5HF.0 size=238863 2023-12-03T11:30:44.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.705+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=UK8IJW1CN5HF tag=3RDQ22XH83W2 type=write size=1287439 2023-12-03T11:30:44.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.706+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=2C1M9PELCFQE type=del size=3193135 2023-12-03T11:30:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.709+0000 7fc553ded780 1 < suggested remove operation key=FP2UZMDKYTD1 tag=2PLAOXGSO5I1 type=del 2023-12-03T11:30:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.710+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=KRYXUNPCOVPE type=del size=1000650 2023-12-03T11:30:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.711+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=FXF7OZTR2C8P type=del 2023-12-03T11:30:44.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.714+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.0 size=1048576 2023-12-03T11:30:44.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.715+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.1 size=1048576 2023-12-03T11:30:44.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.716+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.2 size=587507 2023-12-03T11:30:44.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.716+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=7E1HCVRSJC29 tag=FPKA4736LY44 type=write size=3733235 2023-12-03T11:30:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.718+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=D4SX2EJP6Q7G type=write 2023-12-03T11:30:44.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.719+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=KRJJZD7FDNMF type=del size=2781518 2023-12-03T11:30:44.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.720+0000 7fc553ded780 1 < canceled operation key=WLTMDGRBF4C9 tag=9RZKLABB62TV type=del size=1601958 2023-12-03T11:30:44.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.723+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~JDZQQDDLV8BA.0 size=86082 2023-12-03T11:30:44.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.723+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=JDZQQDDLV8BA tag=H0C773IGZLO3 type=write size=1134658 2023-12-03T11:30:44.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.724+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~PDUTW0DM9SRA.0 size=154410 2023-12-03T11:30:44.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.726+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=PDUTW0DM9SRA tag=SCW41ULTFD61 type=write size=1202986 2023-12-03T11:30:44.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.728+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=956Z10V2FL04 type=del size=1606030 2023-12-03T11:30:44.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.729+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=ONOLFY2T9TAU type=del 2023-12-03T11:30:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.730+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=JW3RB6LEW4G0 type=del size=1674819 2023-12-03T11:30:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.730+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~24OX4574JV1Q.0 size=837223 2023-12-03T11:30:44.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.734+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=24OX4574JV1Q tag=XJDADRZUBQY4 type=write size=1885799 2023-12-03T11:30:44.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.736+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~WWWC7THT17T9.0 size=49680 2023-12-03T11:30:44.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.736+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=WWWC7THT17T9 tag=Q33P3XNNWARI type=write size=1098256 2023-12-03T11:30:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.738+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~KZ2Q7S9CN5MH.0 size=190304 2023-12-03T11:30:44.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.740+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=KZ2Q7S9CN5MH tag=WET240KHXDAS type=write size=1238880 2023-12-03T11:30:44.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.742+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~B2017BWMWJ32.0 size=1048576 2023-12-03T11:30:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.743+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~B2017BWMWJ32.1 size=266745 2023-12-03T11:30:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.743+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=B2017BWMWJ32 tag=NHC3S9Y44SQY type=write size=2363897 2023-12-03T11:30:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.744+0000 7fc553ded780 -1 < failed to complete operation key=C2TPF9Q5Z7XQ tag=YO3SXS0KQUDM type=write size=474617: (22) Invalid argument 2023-12-03T11:30:44.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.745+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=SKFQ7PN81SFR type=write size=145444 2023-12-03T11:30:44.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.747+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~WEXV56T483WT.0 size=66321 2023-12-03T11:30:44.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.749+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=WEXV56T483WT tag=VIRT12581IDY type=write size=1114897 2023-12-03T11:30:44.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.750+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=6CXS888YG76K type=del size=876029 2023-12-03T11:30:44.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.752+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~NFX7W842I7MG.0 size=925991 2023-12-03T11:30:44.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.754+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=NFX7W842I7MG tag=497JZJVS16U8 type=write size=1974567 2023-12-03T11:30:44.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.756+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.0 size=1048576 2023-12-03T11:30:44.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.757+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.1 size=1048576 2023-12-03T11:30:44.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.758+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.2 size=433081 2023-12-03T11:30:44.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.758+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=D0MOYXGNDJ7O tag=WABZJHYBN905 type=write size=3578809 2023-12-03T11:30:44.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.759+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.0 size=1048576 2023-12-03T11:30:44.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.761+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.1 size=1048576 2023-12-03T11:30:44.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.762+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I57T50Z1NYOQ.2 size=243131 2023-12-03T11:30:44.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.764+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=I57T50Z1NYOQ tag=7WMK20UH09RP type=write size=3388859 2023-12-03T11:30:44.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.765+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=TBMWDF0JAHSA type=del size=1791560 2023-12-03T11:30:44.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.766+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=BEML16ZVO5PS type=del 2023-12-03T11:30:44.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.767+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=0AB70E83TCSH type=write size=163995 2023-12-03T11:30:44.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.769+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=BAK3CU0OODN1 type=del 2023-12-03T11:30:44.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.770+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=USME5DK45SR9 type=del size=1004668 2023-12-03T11:30:44.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.771+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~3XPIUZBTB8NT.0 size=1048576 2023-12-03T11:30:44.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.772+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~3XPIUZBTB8NT.1 size=178118 2023-12-03T11:30:44.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.774+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=3XPIUZBTB8NT tag=BU0C46PNT8SX type=write size=2275270 2023-12-03T11:30:44.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.776+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=F9VUXN1Y2T67 type=write size=517346 2023-12-03T11:30:44.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.777+0000 7fc553ded780 1 < canceled operation key=LRN5NYUZ5JT9 tag=2C1M9PELCFQE type=del size=3193135 2023-12-03T11:30:44.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.778+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=3JK1ZQU4K9Y9 type=write size=821292 2023-12-03T11:30:44.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.780+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=KRYXUNPCOVPE type=del 2023-12-03T11:30:44.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.781+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=N3BIFIADSDPR type=del size=3178454 2023-12-03T11:30:44.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.782+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.0 size=1048576 2023-12-03T11:30:44.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.783+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.1 size=1048576 2023-12-03T11:30:44.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.784+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~7E1HCVRSJC29.2 size=587507 2023-12-03T11:30:44.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.786+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=7E1HCVRSJC29 tag=FPKA4736LY44 type=write size=3733235 2023-12-03T11:30:44.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.787+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=M2BQGNJ3P9BD type=del size=3881563 2023-12-03T11:30:44.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.789+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=KRJJZD7FDNMF type=del 2023-12-03T11:30:44.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.791+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~EBJ0NES5NEEO.0 size=1048576 2023-12-03T11:30:44.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.792+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~EBJ0NES5NEEO.1 size=51391 2023-12-03T11:30:44.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.792+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=EBJ0NES5NEEO tag=3VWLT18HUDSZ type=write size=2148543 2023-12-03T11:30:44.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.793+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~JDZQQDDLV8BA.0 size=86082 2023-12-03T11:30:44.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.796+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=JDZQQDDLV8BA tag=H0C773IGZLO3 type=write size=1134658 2023-12-03T11:30:44.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.797+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=TYWX2SLNYOOT type=write size=663338 2023-12-03T11:30:44.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.798+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=956Z10V2FL04 type=del 2023-12-03T11:30:44.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.800+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=DGPYOQHTMNU1 type=del size=2555508 2023-12-03T11:30:44.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.802+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=JW3RB6LEW4G0 type=del size=1674819 2023-12-03T11:30:44.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.803+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=8FKE0M57HFTV type=write size=711082 2023-12-03T11:30:44.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.805+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~WWWC7THT17T9.0 size=49680 2023-12-03T11:30:44.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.807+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=WWWC7THT17T9 tag=Q33P3XNNWARI type=write size=1098256 2023-12-03T11:30:44.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.808+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=HQBDNG65R7II type=del size=751760 2023-12-03T11:30:44.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.809+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~B2017BWMWJ32.0 size=1048576 2023-12-03T11:30:44.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.810+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~B2017BWMWJ32.1 size=266745 2023-12-03T11:30:44.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.814+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=B2017BWMWJ32 tag=NHC3S9Y44SQY type=write size=2363897 2023-12-03T11:30:44.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.817+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.0 size=1048576 2023-12-03T11:30:44.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.818+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.1 size=1048576 2023-12-03T11:30:44.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.819+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.2 size=905784 2023-12-03T11:30:44.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.819+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=DO6LQVWAOASO tag=LVLVG19CXR9P type=write size=4051512 2023-12-03T11:30:44.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.821+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=SKFQ7PN81SFR type=write size=145444 2023-12-03T11:30:44.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.822+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=4NHM8A2167U6 type=del size=2924374 2023-12-03T11:30:44.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.824+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=6CXS888YG76K type=del 2023-12-03T11:30:44.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.825+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=PYNCZSE42RSZ type=del size=3107077 2023-12-03T11:30:44.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.825+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.0 size=1048576 2023-12-03T11:30:44.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.825+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.1 size=1048576 2023-12-03T11:30:44.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.826+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~D0MOYXGNDJ7O.2 size=433081 2023-12-03T11:30:44.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.829+0000 7fc553ded780 1 < canceled multipart upload key=EK7V6U9DSLD0 upload=D0MOYXGNDJ7O tag=WABZJHYBN905 type=write size=3578809 2023-12-03T11:30:44.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.831+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~KFVM6YTYIIMZ.0 size=1008032 2023-12-03T11:30:44.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.831+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=KFVM6YTYIIMZ tag=0LNXSMJDGIMR type=write size=2056608 2023-12-03T11:30:44.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.833+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=TBMWDF0JAHSA type=del 2023-12-03T11:30:44.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.834+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=JPNW39Y4QRV1 type=del size=3483181 2023-12-03T11:30:44.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.837+0000 7fc553ded780 1 < suggested update operation key=WDV4CW863V33 tag=0AB70E83TCSH type=write size=163995 2023-12-03T11:30:44.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.839+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~T5THGYMUP2QX.0 size=666669 2023-12-03T11:30:44.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.839+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=T5THGYMUP2QX tag=Z031EPUDP5Y7 type=write size=1715245 2023-12-03T11:30:44.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.840+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=USME5DK45SR9 type=del 2023-12-03T11:30:44.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.843+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~B2WEQONRJRAR.0 size=1048576 2023-12-03T11:30:44.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.844+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~B2WEQONRJRAR.1 size=466837 2023-12-03T11:30:44.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.844+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=B2WEQONRJRAR tag=AZII4DKBL3E1 type=write size=2563989 2023-12-03T11:30:44.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.845+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=F9VUXN1Y2T67 type=write size=517346 2023-12-03T11:30:44.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.847+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=G7ZSZUAT0KHY type=del size=2803171 2023-12-03T11:30:44.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.848+0000 7fc553ded780 1 < completed write operation key=X741JIRT7RIG tag=3JK1ZQU4K9Y9 type=write size=821292 2023-12-03T11:30:44.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.849+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=QQXMXQZ4L1RN type=del size=2968998 2023-12-03T11:30:44.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.851+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=N3BIFIADSDPR type=del 2023-12-03T11:30:44.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.852+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=PTXS2D38XJKM type=write size=942201 2023-12-03T11:30:44.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.856+0000 7fc553ded780 1 < suggested update operation key=G1OBBD75M5ZQ tag=M2BQGNJ3P9BD type=del size=3881563 2023-12-03T11:30:44.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.857+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=OEWTEBW4CCLG type=del size=3622952 2023-12-03T11:30:44.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.858+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~EBJ0NES5NEEO.0 size=1048576 2023-12-03T11:30:44.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.859+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~EBJ0NES5NEEO.1 size=51391 2023-12-03T11:30:44.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.861+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=EBJ0NES5NEEO tag=3VWLT18HUDSZ type=write size=2148543 2023-12-03T11:30:44.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.863+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=1SQCU0SQNVWF type=del size=2318313 2023-12-03T11:30:44.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.864+0000 7fc553ded780 -1 < failed to complete operation key=G1OBBD75M5ZQ tag=TYWX2SLNYOOT type=write size=663338: (22) Invalid argument 2023-12-03T11:30:44.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.866+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.0 size=1048576 2023-12-03T11:30:44.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.867+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.1 size=1048576 2023-12-03T11:30:44.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.868+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.2 size=195134 2023-12-03T11:30:44.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.868+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=ECA93P0WI5Y9 tag=47YRRFJDYTN3 type=write size=3340862 2023-12-03T11:30:44.870 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.869+0000 7fc553ded780 -1 < failed to complete operation key=4TWM1QM1OHOF tag=DGPYOQHTMNU1 type=del size=2555508: (22) Invalid argument 2023-12-03T11:30:44.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.870+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=3MLPT934HT6Z type=del size=1928839 2023-12-03T11:30:44.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.872+0000 7fc553ded780 1 < completed write operation key=IRMADF9MXXBZ tag=8FKE0M57HFTV type=write size=711082 2023-12-03T11:30:44.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.874+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.0 size=1048576 2023-12-03T11:30:44.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.875+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.1 size=1048576 2023-12-03T11:30:44.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.876+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.2 size=402610 2023-12-03T11:30:44.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.876+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=2C4J0BVBKJJE tag=426AYZTJAXXX type=write size=3548338 2023-12-03T11:30:44.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.877+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=HQBDNG65R7II type=del 2023-12-03T11:30:44.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.879+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~TJHI96IL1TH3.0 size=1048576 2023-12-03T11:30:44.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.880+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~TJHI96IL1TH3.1 size=977457 2023-12-03T11:30:44.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.880+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=TJHI96IL1TH3 tag=JF8NRMIBJUSB type=write size=3074609 2023-12-03T11:30:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.882+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.0 size=1048576 2023-12-03T11:30:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.883+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.1 size=1048576 2023-12-03T11:30:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.884+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~DO6LQVWAOASO.2 size=905784 2023-12-03T11:30:44.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.886+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=DO6LQVWAOASO tag=LVLVG19CXR9P type=write size=4051512 2023-12-03T11:30:44.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.888+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PF2FX4W7B90E.0 size=533995 2023-12-03T11:30:44.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.888+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=PF2FX4W7B90E tag=JF3Z6NMOWDCX type=write size=1582571 2023-12-03T11:30:44.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.890+0000 7fc553ded780 1 < suggested update operation key=JUYA4LXEQ4US tag=4NHM8A2167U6 type=del size=2924374 2023-12-03T11:30:44.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.892+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=5R15ZE6QP7OC type=del size=2129508 2023-12-03T11:30:44.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.893+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=PYNCZSE42RSZ type=del 2023-12-03T11:30:44.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.895+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.0 size=1048576 2023-12-03T11:30:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.896+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.1 size=1048576 2023-12-03T11:30:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.898+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.2 size=873036 2023-12-03T11:30:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.898+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=2Y4S931SO4BQ tag=IWW6XV900WR3 type=write size=4018764 2023-12-03T11:30:44.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.899+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~KFVM6YTYIIMZ.0 size=1008032 2023-12-03T11:30:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.901+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=KFVM6YTYIIMZ tag=0LNXSMJDGIMR type=write size=2056608 2023-12-03T11:30:44.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.903+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=5SEZI4Z1P77Z type=del size=3207658 2023-12-03T11:30:44.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.904+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=JPNW39Y4QRV1 type=del 2023-12-03T11:30:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.905+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=QYG5CTSK1GUU type=del size=108605 2023-12-03T11:30:44.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.906+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~T5THGYMUP2QX.0 size=666669 2023-12-03T11:30:44.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.908+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=T5THGYMUP2QX tag=Z031EPUDP5Y7 type=write size=1715245 2023-12-03T11:30:44.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.910+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=59NT3L3HQGHO type=del size=2193780 2023-12-03T11:30:44.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.911+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~B2WEQONRJRAR.0 size=1048576 2023-12-03T11:30:44.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.912+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~B2WEQONRJRAR.1 size=466837 2023-12-03T11:30:44.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.914+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=B2WEQONRJRAR tag=AZII4DKBL3E1 type=write size=2563989 2023-12-03T11:30:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.916+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OMDK6VSNHP0G.0 size=1048576 2023-12-03T11:30:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.917+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OMDK6VSNHP0G.1 size=666939 2023-12-03T11:30:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.917+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=OMDK6VSNHP0G tag=BDLLGVPRNH5S type=write size=2764091 2023-12-03T11:30:44.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.918+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=G7ZSZUAT0KHY type=del 2023-12-03T11:30:44.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.920+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=KPR3OMTEOLZR type=write size=510148 2023-12-03T11:30:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.922+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=QQXMXQZ4L1RN type=del 2023-12-03T11:30:44.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.923+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=PAZ77NZ73BCC type=del size=3073024 2023-12-03T11:30:44.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.924+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=PTXS2D38XJKM type=write size=942201 2023-12-03T11:30:44.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.927+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.0 size=1048576 2023-12-03T11:30:44.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.928+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.1 size=1048576 2023-12-03T11:30:44.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.928+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.2 size=808201 2023-12-03T11:30:44.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.928+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=7LQZ6DY5DLT1 tag=3NQ21FVIZS16 type=write size=3953929 2023-12-03T11:30:44.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.930+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=OEWTEBW4CCLG type=del 2023-12-03T11:30:44.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.931+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=G39I07PVSISJ type=del size=1941865 2023-12-03T11:30:44.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.932+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=1SQCU0SQNVWF type=del size=2318313 2023-12-03T11:30:44.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.933+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=C0CUD4GZA28W type=del size=13057 2023-12-03T11:30:44.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.935+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.0 size=1048576 2023-12-03T11:30:44.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.936+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.1 size=1048576 2023-12-03T11:30:44.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.937+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~ECA93P0WI5Y9.2 size=195134 2023-12-03T11:30:44.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.939+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=ECA93P0WI5Y9 tag=47YRRFJDYTN3 type=write size=3340862 2023-12-03T11:30:44.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.940+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=BSLSQDD2UUHC type=del size=4082599 2023-12-03T11:30:44.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.941+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=3MLPT934HT6Z type=del 2023-12-03T11:30:44.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.943+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=X63UN2EIK85U type=write size=247996 2023-12-03T11:30:44.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.944+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.0 size=1048576 2023-12-03T11:30:44.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.944+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.1 size=1048576 2023-12-03T11:30:44.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.945+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~2C4J0BVBKJJE.2 size=402610 2023-12-03T11:30:44.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.947+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=2C4J0BVBKJJE tag=426AYZTJAXXX type=write size=3548338 2023-12-03T11:30:44.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.948+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=V81JNJ42312W type=del size=487922 2023-12-03T11:30:44.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.949+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~TJHI96IL1TH3.0 size=1048576 2023-12-03T11:30:44.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.951+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~TJHI96IL1TH3.1 size=977457 2023-12-03T11:30:44.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.953+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=TJHI96IL1TH3 tag=JF8NRMIBJUSB type=write size=3074609 2023-12-03T11:30:44.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.954+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=0TTI22F0RE8M type=write size=541042 2023-12-03T11:30:44.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.956+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~PF2FX4W7B90E.0 size=533995 2023-12-03T11:30:44.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.958+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=PF2FX4W7B90E tag=JF3Z6NMOWDCX type=write size=1582571 2023-12-03T11:30:44.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.959+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=7GYCNKDYY9EI type=write size=772305 2023-12-03T11:30:44.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.960+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=5R15ZE6QP7OC type=del 2023-12-03T11:30:44.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.961+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=TZP3W5WRXEMI type=del size=2710658 2023-12-03T11:30:44.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.963+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.0 size=1048576 2023-12-03T11:30:44.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.964+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.1 size=1048576 2023-12-03T11:30:44.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.965+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~2Y4S931SO4BQ.2 size=873036 2023-12-03T11:30:44.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.967+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=2Y4S931SO4BQ tag=IWW6XV900WR3 type=write size=4018764 2023-12-03T11:30:44.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.968+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=W3UP0DE6FC9U type=del size=4107297 2023-12-03T11:30:44.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.972+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=5SEZI4Z1P77Z type=del 2023-12-03T11:30:44.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.974+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.0 size=1048576 2023-12-03T11:30:44.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.975+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.1 size=1048576 2023-12-03T11:30:44.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.976+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.2 size=285732 2023-12-03T11:30:44.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.976+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=Z7MKDB4SA3YA tag=ZSVPUIE7LSD1 type=write size=3431460 2023-12-03T11:30:44.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.977+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=QYG5CTSK1GUU type=del 2023-12-03T11:30:44.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.980+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.0 size=1048576 2023-12-03T11:30:44.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.981+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.1 size=1048576 2023-12-03T11:30:44.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.982+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.2 size=654490 2023-12-03T11:30:44.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.982+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=GJNVGSR4EZG0 tag=FYX7C6CS9964 type=write size=3800218 2023-12-03T11:30:44.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.984+0000 7fc553ded780 1 < suggested update operation key=JUYA4LXEQ4US tag=59NT3L3HQGHO type=del size=2193780 2023-12-03T11:30:44.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.986+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~0B245CYE9VF7.0 size=1048576 2023-12-03T11:30:44.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.987+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~0B245CYE9VF7.1 size=287460 2023-12-03T11:30:44.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.987+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=0B245CYE9VF7 tag=TS62COM2WWX2 type=write size=2384612 2023-12-03T11:30:44.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.988+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OMDK6VSNHP0G.0 size=1048576 2023-12-03T11:30:44.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.989+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OMDK6VSNHP0G.1 size=666939 2023-12-03T11:30:44.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.991+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=OMDK6VSNHP0G tag=BDLLGVPRNH5S type=write size=2764091 2023-12-03T11:30:44.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.994+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.0 size=1048576 2023-12-03T11:30:44.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.995+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.1 size=1048576 2023-12-03T11:30:44.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.996+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.2 size=70957 2023-12-03T11:30:44.997 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.996+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=GWYYXM2D0RX9 tag=4SDLEEER6ZJC type=write size=3216685 2023-12-03T11:30:44.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.997+0000 7fc553ded780 1 < completed write operation key=C2TPF9Q5Z7XQ tag=KPR3OMTEOLZR type=write size=510148 2023-12-03T11:30:44.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.998+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=QP1X0TYNPMO9 type=del size=1677069 2023-12-03T11:30:45.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:44.999+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=PAZ77NZ73BCC type=del size=3073024 2023-12-03T11:30:45.002 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.001+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.0 size=1048576 2023-12-03T11:30:45.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.002+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.1 size=1048576 2023-12-03T11:30:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.003+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.2 size=495712 2023-12-03T11:30:45.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.003+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=9136FQ3IXZCF tag=UJNC1TH09SJY type=write size=3641440 2023-12-03T11:30:45.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.005+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.0 size=1048576 2023-12-03T11:30:45.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.005+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.1 size=1048576 2023-12-03T11:30:45.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.005+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~7LQZ6DY5DLT1.2 size=808201 2023-12-03T11:30:45.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.008+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=7LQZ6DY5DLT1 tag=3NQ21FVIZS16 type=write size=3953929 2023-12-03T11:30:45.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.010+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~9GFVFHRTUA4D.0 size=505498 2023-12-03T11:30:45.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.010+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=9GFVFHRTUA4D tag=MC4ZU17GB2FX type=write size=1554074 2023-12-03T11:30:45.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.012+0000 7fc553ded780 1 < suggested update operation key=CZP8OQRU641T tag=G39I07PVSISJ type=del size=1941865 2023-12-03T11:30:45.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.014+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~1VS8PVYYHKQR.0 size=936941 2023-12-03T11:30:45.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.014+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=1VS8PVYYHKQR tag=3431GU1A4VVY type=write size=1985517 2023-12-03T11:30:45.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.016+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=C0CUD4GZA28W type=del 2023-12-03T11:30:45.018 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.017+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=10C1SAFFLNIK type=write size=1029695 2023-12-03T11:30:45.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.018+0000 7fc553ded780 1 < canceled operation key=WDV4CW863V33 tag=BSLSQDD2UUHC type=del size=4082599 2023-12-03T11:30:45.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.020+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~S3SZW6JQSPTZ.0 size=1048576 2023-12-03T11:30:45.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.021+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~S3SZW6JQSPTZ.1 size=535297 2023-12-03T11:30:45.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.021+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=S3SZW6JQSPTZ tag=ACNX8PCEXEB6 type=write size=2632449 2023-12-03T11:30:45.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.022+0000 7fc553ded780 1 < completed write operation key=C2TPF9Q5Z7XQ tag=X63UN2EIK85U type=write size=247996 2023-12-03T11:30:45.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.024+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=3VYO1C65B6QS type=del size=2016081 2023-12-03T11:30:45.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.025+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=V81JNJ42312W type=del 2023-12-03T11:30:45.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.027+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~VRUHHKXGMFKC.0 size=1048576 2023-12-03T11:30:45.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.029+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~VRUHHKXGMFKC.1 size=989553 2023-12-03T11:30:45.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.029+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=VRUHHKXGMFKC tag=XZI9MBSRBU2C type=write size=3086705 2023-12-03T11:30:45.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.030+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=0TTI22F0RE8M type=write size=541042 2023-12-03T11:30:45.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.032+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.0 size=1048576 2023-12-03T11:30:45.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.033+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.1 size=1048576 2023-12-03T11:30:45.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.034+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.2 size=928090 2023-12-03T11:30:45.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.034+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=B6IJ2I1Y24B4 tag=NQLODJWGLXGV type=write size=4073818 2023-12-03T11:30:45.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.035+0000 7fc553ded780 -1 < failed to complete operation key=CZP8OQRU641T tag=7GYCNKDYY9EI type=write size=772305: (22) Invalid argument 2023-12-03T11:30:45.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.037+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=CQGUQ0B73KLU type=del size=408522 2023-12-03T11:30:45.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.038+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=TZP3W5WRXEMI type=del 2023-12-03T11:30:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.040+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~N9IT65WS0R7B.0 size=606316 2023-12-03T11:30:45.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.040+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=N9IT65WS0R7B tag=XRA2OJVQGK8V type=write size=1654892 2023-12-03T11:30:45.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.042+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=W3UP0DE6FC9U type=del 2023-12-03T11:30:45.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.044+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OE3DNY68VLZV.0 size=1048576 2023-12-03T11:30:45.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.045+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OE3DNY68VLZV.1 size=522501 2023-12-03T11:30:45.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.045+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=OE3DNY68VLZV tag=8LH10XTNV6DN type=write size=2619653 2023-12-03T11:30:45.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.046+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.0 size=1048576 2023-12-03T11:30:45.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.047+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.1 size=1048576 2023-12-03T11:30:45.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.047+0000 7fc553ded780 1 < canceled part key=_multipart_I3U08QX8MLBD.2~Z7MKDB4SA3YA.2 size=285732 2023-12-03T11:30:45.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.049+0000 7fc553ded780 1 < canceled multipart upload key=I3U08QX8MLBD upload=Z7MKDB4SA3YA tag=ZSVPUIE7LSD1 type=write size=3431460 2023-12-03T11:30:45.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.052+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~SKW5EGCGV7HR.0 size=1048576 2023-12-03T11:30:45.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.053+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~SKW5EGCGV7HR.1 size=251783 2023-12-03T11:30:45.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.053+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=SKW5EGCGV7HR tag=8FPX3VS8WO8U type=write size=2348935 2023-12-03T11:30:45.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.053+0000 7fc553ded780 1 < canceled part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.0 size=1048576 2023-12-03T11:30:45.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.054+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.1 size=1048576 2023-12-03T11:30:45.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.055+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~GJNVGSR4EZG0.2 size=654490 2023-12-03T11:30:45.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.057+0000 7fc553ded780 1 < canceled multipart upload key=ZHK2M78A0SLQ upload=GJNVGSR4EZG0 tag=FYX7C6CS9964 type=write size=3800218 2023-12-03T11:30:45.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.060+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~HA41R1A3LLT0.0 size=220966 2023-12-03T11:30:45.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.060+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=HA41R1A3LLT0 tag=AYCOPJ0NK2WR type=write size=1269542 2023-12-03T11:30:45.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.061+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~0B245CYE9VF7.0 size=1048576 2023-12-03T11:30:45.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.062+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~0B245CYE9VF7.1 size=287460 2023-12-03T11:30:45.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.064+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=0B245CYE9VF7 tag=TS62COM2WWX2 type=write size=2384612 2023-12-03T11:30:45.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.066+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.0 size=1048576 2023-12-03T11:30:45.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.067+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.1 size=1048576 2023-12-03T11:30:45.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.068+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.2 size=381063 2023-12-03T11:30:45.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.068+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=8Q68NGA16DVM tag=YW8RGAT1162T type=write size=3526791 2023-12-03T11:30:45.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.070+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.0 size=1048576 2023-12-03T11:30:45.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.071+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.1 size=1048576 2023-12-03T11:30:45.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.072+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~GWYYXM2D0RX9.2 size=70957 2023-12-03T11:30:45.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.074+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=GWYYXM2D0RX9 tag=4SDLEEER6ZJC type=write size=3216685 2023-12-03T11:30:45.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.076+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=UCBD27ORT01B type=del size=1989114 2023-12-03T11:30:45.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.077+0000 7fc553ded780 1 < canceled operation key=I3U08QX8MLBD tag=QP1X0TYNPMO9 type=del size=1677069 2023-12-03T11:30:45.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.078+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=DHXJZWN79U8F type=del size=2713464 2023-12-03T11:30:45.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.080+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.0 size=1048576 2023-12-03T11:30:45.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.081+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.1 size=1048576 2023-12-03T11:30:45.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.082+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~9136FQ3IXZCF.2 size=495712 2023-12-03T11:30:45.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.084+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=9136FQ3IXZCF tag=UJNC1TH09SJY type=write size=3641440 2023-12-03T11:30:45.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.088+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~R36JSVWCQ1SN.0 size=160579 2023-12-03T11:30:45.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.088+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=R36JSVWCQ1SN tag=8VVAQNWPZHDR type=write size=1209155 2023-12-03T11:30:45.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.090+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~9GFVFHRTUA4D.0 size=505498 2023-12-03T11:30:45.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.092+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=9GFVFHRTUA4D tag=MC4ZU17GB2FX type=write size=1554074 2023-12-03T11:30:45.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.094+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~133V9IZT6H2C.0 size=1048576 2023-12-03T11:30:45.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.095+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~133V9IZT6H2C.1 size=941545 2023-12-03T11:30:45.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.095+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=133V9IZT6H2C tag=YQM000YUX7RR type=write size=3038697 2023-12-03T11:30:45.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.096+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~1VS8PVYYHKQR.0 size=936941 2023-12-03T11:30:45.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.098+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=1VS8PVYYHKQR tag=3431GU1A4VVY type=write size=1985517 2023-12-03T11:30:45.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.099+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=LTOPB9W1Y6RR type=del size=1640150 2023-12-03T11:30:45.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.100+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=10C1SAFFLNIK type=write size=1029695 2023-12-03T11:30:45.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.103+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~6XXJTW44SZM0.0 size=1048576 2023-12-03T11:30:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.103+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~6XXJTW44SZM0.1 size=738864 2023-12-03T11:30:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.103+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=6XXJTW44SZM0 tag=CV886XZHIJEO type=write size=2836016 2023-12-03T11:30:45.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.104+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~S3SZW6JQSPTZ.0 size=1048576 2023-12-03T11:30:45.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.105+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~S3SZW6JQSPTZ.1 size=535297 2023-12-03T11:30:45.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.107+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=S3SZW6JQSPTZ tag=ACNX8PCEXEB6 type=write size=2632449 2023-12-03T11:30:45.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.109+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.0 size=1048576 2023-12-03T11:30:45.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.110+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.1 size=1048576 2023-12-03T11:30:45.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.111+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.2 size=92392 2023-12-03T11:30:45.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.111+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=QC5O8XU97V4R tag=20CZXV0FNIXE type=write size=3238120 2023-12-03T11:30:45.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.113+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=3VYO1C65B6QS type=del size=2016081 2023-12-03T11:30:45.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.115+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=735ZEY6KANTJ type=del size=3800192 2023-12-03T11:30:45.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.115+0000 7fc553ded780 1 < canceled part key=_multipart_588S2ASN9I0W.2~VRUHHKXGMFKC.0 size=1048576 2023-12-03T11:30:45.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.116+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~VRUHHKXGMFKC.1 size=989553 2023-12-03T11:30:45.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.118+0000 7fc553ded780 1 < canceled multipart upload key=588S2ASN9I0W upload=VRUHHKXGMFKC tag=XZI9MBSRBU2C type=write size=3086705 2023-12-03T11:30:45.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.119+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=62KMYLUYKY6D type=write size=183939 2023-12-03T11:30:45.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.120+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.0 size=1048576 2023-12-03T11:30:45.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.121+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.1 size=1048576 2023-12-03T11:30:45.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.122+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~B6IJ2I1Y24B4.2 size=928090 2023-12-03T11:30:45.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.124+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=B6IJ2I1Y24B4 tag=NQLODJWGLXGV type=write size=4073818 2023-12-03T11:30:45.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.125+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=37M4LE194Q4U type=del size=248909 2023-12-03T11:30:45.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.127+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=CQGUQ0B73KLU type=del 2023-12-03T11:30:45.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.129+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~E71AAQ4FYB35.0 size=1048576 2023-12-03T11:30:45.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.130+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~E71AAQ4FYB35.1 size=756001 2023-12-03T11:30:45.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.130+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=E71AAQ4FYB35 tag=6V4RHNN1LEKQ type=write size=2853153 2023-12-03T11:30:45.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.131+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~N9IT65WS0R7B.0 size=606316 2023-12-03T11:30:45.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.133+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=N9IT65WS0R7B tag=XRA2OJVQGK8V type=write size=1654892 2023-12-03T11:30:45.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.134+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=U1EOYTLGMASQ type=del size=683809 2023-12-03T11:30:45.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.135+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OE3DNY68VLZV.0 size=1048576 2023-12-03T11:30:45.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.136+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OE3DNY68VLZV.1 size=522501 2023-12-03T11:30:45.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.138+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=OE3DNY68VLZV tag=8LH10XTNV6DN type=write size=2619653 2023-12-03T11:30:45.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.139+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=RH8720LJH6E2 type=write size=345880 2023-12-03T11:30:45.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.141+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~SKW5EGCGV7HR.0 size=1048576 2023-12-03T11:30:45.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.141+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~SKW5EGCGV7HR.1 size=251783 2023-12-03T11:30:45.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.143+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=SKW5EGCGV7HR tag=8FPX3VS8WO8U type=write size=2348935 2023-12-03T11:30:45.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.144+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=HJ2DYZMD2ZYU type=write size=530614 2023-12-03T11:30:45.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.144+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~HA41R1A3LLT0.0 size=220966 2023-12-03T11:30:45.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.146+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=HA41R1A3LLT0 tag=AYCOPJ0NK2WR type=write size=1269542 2023-12-03T11:30:45.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.147+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=IXW4VKB5UE4U type=del size=1867652 2023-12-03T11:30:45.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.148+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.0 size=1048576 2023-12-03T11:30:45.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.149+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.1 size=1048576 2023-12-03T11:30:45.151 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.150+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~8Q68NGA16DVM.2 size=381063 2023-12-03T11:30:45.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.153+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=8Q68NGA16DVM tag=YW8RGAT1162T type=write size=3526791 2023-12-03T11:30:45.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.155+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~IV8IJOUSN10C.0 size=1048576 2023-12-03T11:30:45.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.156+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~IV8IJOUSN10C.1 size=620420 2023-12-03T11:30:45.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.156+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=IV8IJOUSN10C tag=BR4IQ1FAXRXH type=write size=2717572 2023-12-03T11:30:45.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.157+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=UCBD27ORT01B type=del 2023-12-03T11:30:45.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.158+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=WD7QGC74WOOG type=write size=1016175 2023-12-03T11:30:45.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.159+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=DHXJZWN79U8F type=del 2023-12-03T11:30:45.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.160+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=3UQ3BYH5XX1S type=del size=3169388 2023-12-03T11:30:45.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.162+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~R36JSVWCQ1SN.0 size=160579 2023-12-03T11:30:45.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.164+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=R36JSVWCQ1SN tag=8VVAQNWPZHDR type=write size=1209155 2023-12-03T11:30:45.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.166+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.0 size=1048576 2023-12-03T11:30:45.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.167+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.1 size=1048576 2023-12-03T11:30:45.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.168+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.2 size=143336 2023-12-03T11:30:45.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.168+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=2QPBE222C0P4 tag=CHY5OUUSTF2C type=write size=3289064 2023-12-03T11:30:45.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.170+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~133V9IZT6H2C.0 size=1048576 2023-12-03T11:30:45.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.171+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~133V9IZT6H2C.1 size=941545 2023-12-03T11:30:45.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.172+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=133V9IZT6H2C tag=YQM000YUX7RR type=write size=3038697 2023-12-03T11:30:45.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.174+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=8CD9K3YLS43J type=del size=2041663 2023-12-03T11:30:45.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.176+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=LTOPB9W1Y6RR type=del size=1640150 2023-12-03T11:30:45.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.177+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=JNX8RNKDWWGH type=del size=1885270 2023-12-03T11:30:45.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.179+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~6XXJTW44SZM0.0 size=1048576 2023-12-03T11:30:45.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.186+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~6XXJTW44SZM0.1 size=738864 2023-12-03T11:30:45.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.188+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=6XXJTW44SZM0 tag=CV886XZHIJEO type=write size=2836016 2023-12-03T11:30:45.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.189+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=DK3YQG99OQDL type=del size=695157 2023-12-03T11:30:45.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.191+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.0 size=1048576 2023-12-03T11:30:45.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.192+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.1 size=1048576 2023-12-03T11:30:45.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.193+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~QC5O8XU97V4R.2 size=92392 2023-12-03T11:30:45.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.195+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=QC5O8XU97V4R tag=20CZXV0FNIXE type=write size=3238120 2023-12-03T11:30:45.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.197+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.0 size=1048576 2023-12-03T11:30:45.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.198+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.1 size=1048576 2023-12-03T11:30:45.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.199+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.2 size=638882 2023-12-03T11:30:45.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.199+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=Q3X03PSD9VXF tag=GME1A137ATI8 type=write size=3784610 2023-12-03T11:30:45.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.200+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=735ZEY6KANTJ type=del 2023-12-03T11:30:45.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.202+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.0 size=1048576 2023-12-03T11:30:45.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.205+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.1 size=1048576 2023-12-03T11:30:45.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.206+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.2 size=308541 2023-12-03T11:30:45.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.206+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=QMNQ6LTZN6JL tag=58R06M6Y7DFW type=write size=3454269 2023-12-03T11:30:45.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.207+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=62KMYLUYKY6D type=write size=183939 2023-12-03T11:30:45.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.208+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=QVRH9BIZLUYI type=del size=3117125 2023-12-03T11:30:45.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.210+0000 7fc553ded780 1 < suggested remove operation key=0H5OZMBMLQFJ tag=37M4LE194Q4U type=del 2023-12-03T11:30:45.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.212+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=EG9DNHM16YUU type=del size=144688 2023-12-03T11:30:45.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.213+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~E71AAQ4FYB35.0 size=1048576 2023-12-03T11:30:45.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.214+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~E71AAQ4FYB35.1 size=756001 2023-12-03T11:30:45.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.216+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=E71AAQ4FYB35 tag=6V4RHNN1LEKQ type=write size=2853153 2023-12-03T11:30:45.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.217+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=OEJC29IA20UH type=write size=98104 2023-12-03T11:30:45.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.218+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=U1EOYTLGMASQ type=del 2023-12-03T11:30:45.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.221+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~0U1KVSH50ZQC.0 size=769664 2023-12-03T11:30:45.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.221+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=0U1KVSH50ZQC tag=W1CJJJ33A27C type=write size=1818240 2023-12-03T11:30:45.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.222+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=RH8720LJH6E2 type=write size=345880 2023-12-03T11:30:45.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.224+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~N8YUVDT33878.0 size=1048576 2023-12-03T11:30:45.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.225+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~N8YUVDT33878.1 size=223658 2023-12-03T11:30:45.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.225+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=N8YUVDT33878 tag=AU90308VMRF5 type=write size=2320810 2023-12-03T11:30:45.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.226+0000 7fc553ded780 1 < completed write operation key=G1OBBD75M5ZQ tag=HJ2DYZMD2ZYU type=write size=530614 2023-12-03T11:30:45.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.228+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=SRSEZKNKNDFQ type=del size=1275335 2023-12-03T11:30:45.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.229+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=IXW4VKB5UE4U type=del 2023-12-03T11:30:45.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.230+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=63M6YOQQCB5B type=del size=2646266 2023-12-03T11:30:45.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.231+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~IV8IJOUSN10C.0 size=1048576 2023-12-03T11:30:45.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.232+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~IV8IJOUSN10C.1 size=620420 2023-12-03T11:30:45.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.234+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=IV8IJOUSN10C tag=BR4IQ1FAXRXH type=write size=2717572 2023-12-03T11:30:45.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.235+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=4JN76EOLQ1RT type=del size=3048638 2023-12-03T11:30:45.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.237+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=WD7QGC74WOOG type=write size=1016175 2023-12-03T11:30:45.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.238+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=8DNF1LRWA8R7 type=write size=661562 2023-12-03T11:30:45.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.239+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=3UQ3BYH5XX1S type=del 2023-12-03T11:30:45.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.240+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=FU0JGJ9ZPK4N type=del size=362641 2023-12-03T11:30:45.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.241+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.0 size=1048576 2023-12-03T11:30:45.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.242+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.1 size=1048576 2023-12-03T11:30:45.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.242+0000 7fc553ded780 1 < canceled part key=_multipart_706CET5PSDEG.2~2QPBE222C0P4.2 size=143336 2023-12-03T11:30:45.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.244+0000 7fc553ded780 1 < canceled multipart upload key=706CET5PSDEG upload=2QPBE222C0P4 tag=CHY5OUUSTF2C type=write size=3289064 2023-12-03T11:30:45.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.246+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=E5FOK3WYIJR2 type=del size=3550298 2023-12-03T11:30:45.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.247+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=8CD9K3YLS43J type=del 2023-12-03T11:30:45.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.248+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=D2BH4KBACDEG type=write size=430785 2023-12-03T11:30:45.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.249+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=JNX8RNKDWWGH type=del 2023-12-03T11:30:45.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.255+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~2CK5H7P8GSQH.0 size=349026 2023-12-03T11:30:45.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.255+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=2CK5H7P8GSQH tag=6R6902WGXBYH type=write size=1397602 2023-12-03T11:30:45.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.257+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=DK3YQG99OQDL type=del 2023-12-03T11:30:45.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.259+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=PR3VFVIZA219 type=del size=446367 2023-12-03T11:30:45.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.260+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.0 size=1048576 2023-12-03T11:30:45.262 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.261+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.1 size=1048576 2023-12-03T11:30:45.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.262+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Q3X03PSD9VXF.2 size=638882 2023-12-03T11:30:45.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.265+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=Q3X03PSD9VXF tag=GME1A137ATI8 type=write size=3784610 2023-12-03T11:30:45.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.266+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=I3HK1CQWNBLX type=del size=155281 2023-12-03T11:30:45.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.266+0000 7fc553ded780 1 < canceled part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.0 size=1048576 2023-12-03T11:30:45.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.267+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.1 size=1048576 2023-12-03T11:30:45.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.268+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~QMNQ6LTZN6JL.2 size=308541 2023-12-03T11:30:45.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.270+0000 7fc553ded780 1 < canceled multipart upload key=FP2UZMDKYTD1 upload=QMNQ6LTZN6JL tag=58R06M6Y7DFW type=write size=3454269 2023-12-03T11:30:45.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.272+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5RGO0PF2V4VO.0 size=1048576 2023-12-03T11:30:45.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.273+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~5RGO0PF2V4VO.1 size=496516 2023-12-03T11:30:45.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.273+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=5RGO0PF2V4VO tag=OAY94U14342I type=write size=2593668 2023-12-03T11:30:45.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.275+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=QVRH9BIZLUYI type=del 2023-12-03T11:30:45.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.277+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9OQTT0LCMUCF.0 size=1048576 2023-12-03T11:30:45.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.278+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9OQTT0LCMUCF.1 size=255006 2023-12-03T11:30:45.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.278+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=9OQTT0LCMUCF tag=5D3MPXGDO3E9 type=write size=2352158 2023-12-03T11:30:45.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.280+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=EG9DNHM16YUU type=del 2023-12-03T11:30:45.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.283+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~W9V2HTI64HHG.0 size=719615 2023-12-03T11:30:45.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.283+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=W9V2HTI64HHG tag=TVRMDPOMGKQG type=write size=1768191 2023-12-03T11:30:45.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.284+0000 7fc553ded780 1 < completed write operation key=TQRZT37AB9WC tag=OEJC29IA20UH type=write size=98104 2023-12-03T11:30:45.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.286+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~7FCN1JXXQWVM.0 size=525734 2023-12-03T11:30:45.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.286+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=7FCN1JXXQWVM tag=FEOLWU861GLD type=write size=1574310 2023-12-03T11:30:45.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.286+0000 7fc553ded780 1 < canceled part key=_multipart_6OW429S51UWN.2~0U1KVSH50ZQC.0 size=769664 2023-12-03T11:30:45.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.288+0000 7fc553ded780 1 < canceled multipart upload key=6OW429S51UWN upload=0U1KVSH50ZQC tag=W1CJJJ33A27C type=write size=1818240 2023-12-03T11:30:45.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.291+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.0 size=1048576 2023-12-03T11:30:45.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.292+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.1 size=1048576 2023-12-03T11:30:45.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.293+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.2 size=347748 2023-12-03T11:30:45.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.293+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=SOHNDAYAKHZA tag=KC6MLVSVP3HC type=write size=3493476 2023-12-03T11:30:45.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.294+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~N8YUVDT33878.0 size=1048576 2023-12-03T11:30:45.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.295+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~N8YUVDT33878.1 size=223658 2023-12-03T11:30:45.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.298+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=N8YUVDT33878 tag=AU90308VMRF5 type=write size=2320810 2023-12-03T11:30:45.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.299+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=TJDLR0EBGPGG type=del size=1239651 2023-12-03T11:30:45.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.300+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=SRSEZKNKNDFQ type=del 2023-12-03T11:30:45.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.302+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=KZ5MP2UHXF60 type=del size=889639 2023-12-03T11:30:45.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.303+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=63M6YOQQCB5B type=del 2023-12-03T11:30:45.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.305+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~F3Q98BRDEYNP.0 size=1048576 2023-12-03T11:30:45.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.306+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~F3Q98BRDEYNP.1 size=833612 2023-12-03T11:30:45.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.306+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=F3Q98BRDEYNP tag=A29UBVKBKQSN type=write size=2930764 2023-12-03T11:30:45.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.308+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=4JN76EOLQ1RT type=del 2023-12-03T11:30:45.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.309+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=5A2HCSQ3RJKO type=write size=694678 2023-12-03T11:30:45.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.311+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=8DNF1LRWA8R7 type=write 2023-12-03T11:30:45.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.312+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=HU8Y1N8KHOFE type=del size=1503374 2023-12-03T11:30:45.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.314+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=FU0JGJ9ZPK4N type=del 2023-12-03T11:30:45.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.316+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.0 size=1048576 2023-12-03T11:30:45.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.317+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.1 size=1048576 2023-12-03T11:30:45.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.318+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.2 size=201854 2023-12-03T11:30:45.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.318+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=M9BB4ONIF8B7 tag=0PY06RWF7K73 type=write size=3347582 2023-12-03T11:30:45.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.319+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=E5FOK3WYIJR2 type=del 2023-12-03T11:30:45.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.320+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=15MOFDNICFZ1 type=del size=2321522 2023-12-03T11:30:45.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.321+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=D2BH4KBACDEG type=write size=430785 2023-12-03T11:30:45.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.323+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=9ZZXLKLTIQ3R type=del size=105835 2023-12-03T11:30:45.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.324+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~2CK5H7P8GSQH.0 size=349026 2023-12-03T11:30:45.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.327+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=2CK5H7P8GSQH tag=6R6902WGXBYH type=write size=1397602 2023-12-03T11:30:45.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.328+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=FN73YOHEN4NO type=del size=3652496 2023-12-03T11:30:45.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.332+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=PR3VFVIZA219 type=del 2023-12-03T11:30:45.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.333+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=G234RB74UNYX type=del size=2316658 2023-12-03T11:30:45.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.336+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=I3HK1CQWNBLX type=del 2023-12-03T11:30:45.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.337+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=75K8O9MIORRD type=write size=455907 2023-12-03T11:30:45.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.338+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~5RGO0PF2V4VO.0 size=1048576 2023-12-03T11:30:45.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.339+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~5RGO0PF2V4VO.1 size=496516 2023-12-03T11:30:45.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.341+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=5RGO0PF2V4VO tag=OAY94U14342I type=write size=2593668 2023-12-03T11:30:45.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.343+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=I014IDMAE6RA type=del size=2309091 2023-12-03T11:30:45.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.344+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9OQTT0LCMUCF.0 size=1048576 2023-12-03T11:30:45.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.344+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~9OQTT0LCMUCF.1 size=255006 2023-12-03T11:30:45.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.346+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=9OQTT0LCMUCF tag=5D3MPXGDO3E9 type=write size=2352158 2023-12-03T11:30:45.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.347+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=MQZ4HDZ216R1 type=write size=196281 2023-12-03T11:30:45.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.349+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~W9V2HTI64HHG.0 size=719615 2023-12-03T11:30:45.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.351+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=W9V2HTI64HHG tag=TVRMDPOMGKQG type=write size=1768191 2023-12-03T11:30:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.354+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~BWVM7XFVCF01.0 size=1022800 2023-12-03T11:30:45.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.354+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=BWVM7XFVCF01 tag=2TH74TOHZGRQ type=write size=2071376 2023-12-03T11:30:45.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.355+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~7FCN1JXXQWVM.0 size=525734 2023-12-03T11:30:45.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.358+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=7FCN1JXXQWVM tag=FEOLWU861GLD type=write size=1574310 2023-12-03T11:30:45.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.359+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=ORAH5C6KIL6S type=del size=710955 2023-12-03T11:30:45.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.360+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.0 size=1048576 2023-12-03T11:30:45.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.361+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.1 size=1048576 2023-12-03T11:30:45.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.362+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~SOHNDAYAKHZA.2 size=347748 2023-12-03T11:30:45.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.364+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=SOHNDAYAKHZA tag=KC6MLVSVP3HC type=write size=3493476 2023-12-03T11:30:45.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.365+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=07U5CHFHM9CZ type=del size=619785 2023-12-03T11:30:45.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.367+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=TJDLR0EBGPGG type=del 2023-12-03T11:30:45.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.369+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~8G6YK3HQJGPS.0 size=132364 2023-12-03T11:30:45.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.369+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=8G6YK3HQJGPS tag=H95KR44O0GAG type=write size=1180940 2023-12-03T11:30:45.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.370+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=KZ5MP2UHXF60 type=del 2023-12-03T11:30:45.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.371+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=K78D1DXIPXXQ type=write size=799982 2023-12-03T11:30:45.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.372+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~F3Q98BRDEYNP.0 size=1048576 2023-12-03T11:30:45.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.373+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~F3Q98BRDEYNP.1 size=833612 2023-12-03T11:30:45.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.375+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=F3Q98BRDEYNP tag=A29UBVKBKQSN type=write size=2930764 2023-12-03T11:30:45.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.376+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=A2GUYKZX5K09 type=write size=1045822 2023-12-03T11:30:45.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.378+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=5A2HCSQ3RJKO type=write size=694678 2023-12-03T11:30:45.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.379+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=JSAC81FENI1B type=del size=707011 2023-12-03T11:30:45.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.380+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=HU8Y1N8KHOFE type=del 2023-12-03T11:30:45.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.382+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~SBKY5SZBMVIT.0 size=821111 2023-12-03T11:30:45.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.382+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=SBKY5SZBMVIT tag=FGBQK9EZ7AGV type=write size=1869687 2023-12-03T11:30:45.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.383+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.0 size=1048576 2023-12-03T11:30:45.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.384+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.1 size=1048576 2023-12-03T11:30:45.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.385+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~M9BB4ONIF8B7.2 size=201854 2023-12-03T11:30:45.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.388+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=M9BB4ONIF8B7 tag=0PY06RWF7K73 type=write size=3347582 2023-12-03T11:30:45.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.390+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~5DN8RS8G6QSH.0 size=298313 2023-12-03T11:30:45.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.390+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=5DN8RS8G6QSH tag=N7XUG5L2OLFZ type=write size=1346889 2023-12-03T11:30:45.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.391+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=15MOFDNICFZ1 type=del 2023-12-03T11:30:45.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.392+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=WDDMC96V8KGA type=del size=637560 2023-12-03T11:30:45.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.393+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=9ZZXLKLTIQ3R type=del 2023-12-03T11:30:45.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.395+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=C1RRGOKKAXT2 type=del size=3228111 2023-12-03T11:30:45.398 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.397+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=FN73YOHEN4NO type=del size=3652496 2023-12-03T11:30:45.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.399+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~DNLPAH28SR23.0 size=1048576 2023-12-03T11:30:45.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.400+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~DNLPAH28SR23.1 size=275233 2023-12-03T11:30:45.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.400+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=DNLPAH28SR23 tag=SFSPK7ONKSJ2 type=write size=2372385 2023-12-03T11:30:45.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.401+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=G234RB74UNYX type=del 2023-12-03T11:30:45.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.404+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~Y8O82HOWM10P.0 size=698817 2023-12-03T11:30:45.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.404+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=Y8O82HOWM10P tag=YTKLM55OKNLS type=write size=1747393 2023-12-03T11:30:45.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.405+0000 7fc553ded780 1 < completed write operation key=LRN5NYUZ5JT9 tag=75K8O9MIORRD type=write size=455907 2023-12-03T11:30:45.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.407+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PYOVGPSETC7L.0 size=1048576 2023-12-03T11:30:45.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.408+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~PYOVGPSETC7L.1 size=266533 2023-12-03T11:30:45.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.408+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=PYOVGPSETC7L tag=W33KFCTL0822 type=write size=2363685 2023-12-03T11:30:45.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.409+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=I014IDMAE6RA type=del size=2309091 2023-12-03T11:30:45.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.410+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=IVM2VM912GG0 type=write size=661309 2023-12-03T11:30:45.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.412+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=MQZ4HDZ216R1 type=write size=196281 2023-12-03T11:30:45.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.413+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=R2ITXXPJ3AA2 type=del size=3420195 2023-12-03T11:30:45.415 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.414+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~BWVM7XFVCF01.0 size=1022800 2023-12-03T11:30:45.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.416+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=BWVM7XFVCF01 tag=2TH74TOHZGRQ type=write size=2071376 2023-12-03T11:30:45.419 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.418+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.0 size=1048576 2023-12-03T11:30:45.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.419+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.1 size=1048576 2023-12-03T11:30:45.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.420+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.2 size=833775 2023-12-03T11:30:45.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.420+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=FPM57ATOEE9X tag=2ODZX74X7XY7 type=write size=3979503 2023-12-03T11:30:45.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.423+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=ORAH5C6KIL6S type=del size=710955 2023-12-03T11:30:45.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.424+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=TKRTSP99FK9Y type=del size=581697 2023-12-03T11:30:45.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.425+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=07U5CHFHM9CZ type=del 2023-12-03T11:30:45.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.427+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~V7027Q00GAVZ.0 size=991062 2023-12-03T11:30:45.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.427+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=V7027Q00GAVZ tag=PSE20PXMB1PT type=write size=2039638 2023-12-03T11:30:45.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.428+0000 7fc553ded780 1 < canceled part key=_multipart_ZHK2M78A0SLQ.2~8G6YK3HQJGPS.0 size=132364 2023-12-03T11:30:45.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.430+0000 7fc553ded780 1 < canceled multipart upload key=ZHK2M78A0SLQ upload=8G6YK3HQJGPS tag=H95KR44O0GAG type=write size=1180940 2023-12-03T11:30:45.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.431+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=HJ2UCRUQYF78 type=del size=1668955 2023-12-03T11:30:45.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.432+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=K78D1DXIPXXQ type=write size=799982 2023-12-03T11:30:45.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.433+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=7ZKWKBUNJKPT type=del size=2524070 2023-12-03T11:30:45.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.435+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=A2GUYKZX5K09 type=write size=1045822 2023-12-03T11:30:45.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.436+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=IFZSUQE0UTMM type=del size=2921922 2023-12-03T11:30:45.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.437+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=JSAC81FENI1B type=del 2023-12-03T11:30:45.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.439+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~PH1RW63UJ6YZ.0 size=1048576 2023-12-03T11:30:45.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.440+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~PH1RW63UJ6YZ.1 size=255744 2023-12-03T11:30:45.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.440+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=PH1RW63UJ6YZ tag=8O6F6795IBCW type=write size=2352896 2023-12-03T11:30:45.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.441+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~SBKY5SZBMVIT.0 size=821111 2023-12-03T11:30:45.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.443+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=SBKY5SZBMVIT tag=FGBQK9EZ7AGV type=write size=1869687 2023-12-03T11:30:45.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.445+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=9CI8MV0UH00E type=write size=38144 2023-12-03T11:30:45.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.447+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~5DN8RS8G6QSH.0 size=298313 2023-12-03T11:30:45.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.449+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=5DN8RS8G6QSH tag=N7XUG5L2OLFZ type=write size=1346889 2023-12-03T11:30:45.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.451+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=NEM21B0YO2PT type=del size=3924059 2023-12-03T11:30:45.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.452+0000 7fc553ded780 -1 < failed to complete operation key=FC4WU6SMNEWV tag=WDDMC96V8KGA type=del size=637560: (22) Invalid argument 2023-12-03T11:30:45.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.453+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=YOZ2I6DU3DBP type=del size=439427 2023-12-03T11:30:45.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.454+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=C1RRGOKKAXT2 type=del 2023-12-03T11:30:45.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.455+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=BF135LTKH6AT type=del size=1989474 2023-12-03T11:30:45.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.456+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~DNLPAH28SR23.0 size=1048576 2023-12-03T11:30:45.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.457+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~DNLPAH28SR23.1 size=275233 2023-12-03T11:30:45.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.459+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=DNLPAH28SR23 tag=SFSPK7ONKSJ2 type=write size=2372385 2023-12-03T11:30:45.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.461+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=585341IWIYFH type=del size=1658175 2023-12-03T11:30:45.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.462+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~Y8O82HOWM10P.0 size=698817 2023-12-03T11:30:45.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.464+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=Y8O82HOWM10P tag=YTKLM55OKNLS type=write size=1747393 2023-12-03T11:30:45.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.465+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=LATBOMQXPLZF type=write size=643394 2023-12-03T11:30:45.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.466+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PYOVGPSETC7L.0 size=1048576 2023-12-03T11:30:45.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.467+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~PYOVGPSETC7L.1 size=266533 2023-12-03T11:30:45.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.469+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=PYOVGPSETC7L tag=W33KFCTL0822 type=write size=2363685 2023-12-03T11:30:45.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.470+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=XGLHQOFG96MU type=del size=3439938 2023-12-03T11:30:45.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.473+0000 7fc553ded780 1 < suggested remove operation key=JUYA4LXEQ4US tag=IVM2VM912GG0 type=write 2023-12-03T11:30:45.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.474+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=FIX5V9Z01B2F type=del size=3437866 2023-12-03T11:30:45.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.475+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=R2ITXXPJ3AA2 type=del 2023-12-03T11:30:45.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.476+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=EAJY2QIAJOH6 type=del size=1765261 2023-12-03T11:30:45.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.477+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.0 size=1048576 2023-12-03T11:30:45.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.478+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.1 size=1048576 2023-12-03T11:30:45.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.479+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~FPM57ATOEE9X.2 size=833775 2023-12-03T11:30:45.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.481+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=FPM57ATOEE9X tag=2ODZX74X7XY7 type=write size=3979503 2023-12-03T11:30:45.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.482+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=1AF727MZD5MQ type=del size=3986742 2023-12-03T11:30:45.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.484+0000 7fc553ded780 1 < canceled operation key=FP2UZMDKYTD1 tag=TKRTSP99FK9Y type=del size=581697 2023-12-03T11:30:45.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.486+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~LU02CGNOPOY4.0 size=1048576 2023-12-03T11:30:45.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.487+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~LU02CGNOPOY4.1 size=586012 2023-12-03T11:30:45.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.487+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=LU02CGNOPOY4 tag=DIWVSUK0FBF7 type=write size=2683164 2023-12-03T11:30:45.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.488+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~V7027Q00GAVZ.0 size=991062 2023-12-03T11:30:45.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.490+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=V7027Q00GAVZ tag=PSE20PXMB1PT type=write size=2039638 2023-12-03T11:30:45.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.491+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=4IDNEBA9X133 type=del size=2102120 2023-12-03T11:30:45.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.492+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=HJ2UCRUQYF78 type=del 2023-12-03T11:30:45.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.493+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=IALSZJOW5W5M type=del size=1383163 2023-12-03T11:30:45.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.494+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=7ZKWKBUNJKPT type=del 2023-12-03T11:30:45.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.496+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~SJGNN1A5JGQ5.0 size=1048576 2023-12-03T11:30:45.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.497+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~SJGNN1A5JGQ5.1 size=385185 2023-12-03T11:30:45.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.497+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=SJGNN1A5JGQ5 tag=5P2R75JLJGLC type=write size=2482337 2023-12-03T11:30:45.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.498+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=IFZSUQE0UTMM type=del 2023-12-03T11:30:45.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.501+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.0 size=1048576 2023-12-03T11:30:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.502+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.1 size=1048576 2023-12-03T11:30:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.503+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.2 size=856756 2023-12-03T11:30:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.503+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=Q6ODPZ502714 tag=5I535VA32HKW type=write size=4002484 2023-12-03T11:30:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.503+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~PH1RW63UJ6YZ.0 size=1048576 2023-12-03T11:30:45.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.504+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~PH1RW63UJ6YZ.1 size=255744 2023-12-03T11:30:45.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.506+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=PH1RW63UJ6YZ tag=8O6F6795IBCW type=write size=2352896 2023-12-03T11:30:45.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.507+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=MXITZFS3LFT5 type=write size=863362 2023-12-03T11:30:45.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.508+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=9CI8MV0UH00E type=write size=38144 2023-12-03T11:30:45.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.509+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=KQS0KBWB0FJR type=del size=3533885 2023-12-03T11:30:45.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.510+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=NEM21B0YO2PT type=del 2023-12-03T11:30:45.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.512+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=SXN2DU2XHRC2 type=del size=3011124 2023-12-03T11:30:45.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.513+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=YOZ2I6DU3DBP type=del 2023-12-03T11:30:45.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.514+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=JDS4R68CW09Y type=del size=1967838 2023-12-03T11:30:45.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.515+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=BF135LTKH6AT type=del 2023-12-03T11:30:45.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.516+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=2JQGUD6X66A3 type=del size=810174 2023-12-03T11:30:45.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.519+0000 7fc553ded780 1 < suggested update operation key=FP2UZMDKYTD1 tag=585341IWIYFH type=del size=1658175 2023-12-03T11:30:45.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.521+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.0 size=1048576 2023-12-03T11:30:45.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.522+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.1 size=1048576 2023-12-03T11:30:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.523+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.2 size=343426 2023-12-03T11:30:45.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.523+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=81ITO6NCGQYU tag=W0YPCDH2K71V type=write size=3489154 2023-12-03T11:30:45.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.524+0000 7fc553ded780 1 < canceled operation key=I3U08QX8MLBD tag=LATBOMQXPLZF type=write size=643394 2023-12-03T11:30:45.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.525+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=64IK5VLPD2VA type=write size=954913 2023-12-03T11:30:45.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.526+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=XGLHQOFG96MU type=del 2023-12-03T11:30:45.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.527+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=IT2B940RK0LA type=del size=1666899 2023-12-03T11:30:45.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.529+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=FIX5V9Z01B2F type=del 2023-12-03T11:30:45.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.530+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=ECZY6V7E7Q5S type=del size=1724425 2023-12-03T11:30:45.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.531+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=EAJY2QIAJOH6 type=del 2023-12-03T11:30:45.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.533+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~JCY5AP1RUPTJ.0 size=609884 2023-12-03T11:30:45.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.533+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=JCY5AP1RUPTJ tag=BLTS7NIM6BK5 type=write size=1658460 2023-12-03T11:30:45.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.534+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=1AF727MZD5MQ type=del 2023-12-03T11:30:45.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.536+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.0 size=1048576 2023-12-03T11:30:45.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.537+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.1 size=1048576 2023-12-03T11:30:45.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.538+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.2 size=755740 2023-12-03T11:30:45.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.538+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=IFO0ELQSQG97 tag=ZHILJ316GA4O type=write size=3901468 2023-12-03T11:30:45.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.538+0000 7fc553ded780 1 < canceled part key=_multipart_TQRZT37AB9WC.2~LU02CGNOPOY4.0 size=1048576 2023-12-03T11:30:45.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.539+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~LU02CGNOPOY4.1 size=586012 2023-12-03T11:30:45.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.541+0000 7fc553ded780 1 < canceled multipart upload key=TQRZT37AB9WC upload=LU02CGNOPOY4 tag=DIWVSUK0FBF7 type=write size=2683164 2023-12-03T11:30:45.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.542+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=MNUJ26OTYF24 type=del size=2862147 2023-12-03T11:30:45.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.544+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=4IDNEBA9X133 type=del 2023-12-03T11:30:45.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.545+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=I32ZH4F2UDM9 type=del size=1499355 2023-12-03T11:30:45.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.546+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=IALSZJOW5W5M type=del 2023-12-03T11:30:45.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.547+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=DJX4X9RXPTU6 type=del size=1886395 2023-12-03T11:30:45.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.548+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~SJGNN1A5JGQ5.0 size=1048576 2023-12-03T11:30:45.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.548+0000 7fc553ded780 1 < canceled part key=_multipart_ZHK2M78A0SLQ.2~SJGNN1A5JGQ5.1 size=385185 2023-12-03T11:30:45.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.550+0000 7fc553ded780 1 < canceled multipart upload key=ZHK2M78A0SLQ upload=SJGNN1A5JGQ5 tag=5P2R75JLJGLC type=write size=2482337 2023-12-03T11:30:45.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.551+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=8N6KBRVCKM5T type=del size=3525965 2023-12-03T11:30:45.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.553+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.0 size=1048576 2023-12-03T11:30:45.554 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.553+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.1 size=1048576 2023-12-03T11:30:45.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.554+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~Q6ODPZ502714.2 size=856756 2023-12-03T11:30:45.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.558+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=Q6ODPZ502714 tag=5I535VA32HKW type=write size=4002484 2023-12-03T11:30:45.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.559+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=IMBLW657FPRJ type=del size=2780820 2023-12-03T11:30:45.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.561+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=MXITZFS3LFT5 type=write 2023-12-03T11:30:45.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.563+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~FFFAK1T25KAT.0 size=1048576 2023-12-03T11:30:45.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.564+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~FFFAK1T25KAT.1 size=746689 2023-12-03T11:30:45.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.564+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=FFFAK1T25KAT tag=HNOZDF00QG80 type=write size=2843841 2023-12-03T11:30:45.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.566+0000 7fc553ded780 1 < suggested remove operation key=WDV4CW863V33 tag=KQS0KBWB0FJR type=del 2023-12-03T11:30:45.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.568+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=O9XYOEUFF2EB type=del size=3465451 2023-12-03T11:30:45.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.570+0000 7fc553ded780 1 < suggested remove operation key=C2TPF9Q5Z7XQ tag=SXN2DU2XHRC2 type=del 2023-12-03T11:30:45.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.571+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=H34BNK6O9B2T type=del size=4146603 2023-12-03T11:30:45.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.572+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=JDS4R68CW09Y type=del 2023-12-03T11:30:45.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.573+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=8K34VW59K66J type=del size=2726233 2023-12-03T11:30:45.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.575+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=2JQGUD6X66A3 type=del size=810174: (22) Invalid argument 2023-12-03T11:30:45.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.576+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=H7WU6WWI0XED type=del size=2882121 2023-12-03T11:30:45.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.577+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.0 size=1048576 2023-12-03T11:30:45.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.578+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.1 size=1048576 2023-12-03T11:30:45.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.579+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~81ITO6NCGQYU.2 size=343426 2023-12-03T11:30:45.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.581+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=81ITO6NCGQYU tag=W0YPCDH2K71V type=write size=3489154 2023-12-03T11:30:45.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.583+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~KDQZY2FZ8V6E.0 size=924358 2023-12-03T11:30:45.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.583+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=KDQZY2FZ8V6E tag=OS3UOS9TRM1Y type=write size=1972934 2023-12-03T11:30:45.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.584+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=64IK5VLPD2VA type=write size=954913 2023-12-03T11:30:45.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.586+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=HAMC8JZAQI8Z type=del size=3610810 2023-12-03T11:30:45.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.587+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=IT2B940RK0LA type=del 2023-12-03T11:30:45.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.589+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~K12F09SRBQPQ.0 size=305376 2023-12-03T11:30:45.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.589+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=K12F09SRBQPQ tag=0G5EUCZB7VJN type=write size=1353952 2023-12-03T11:30:45.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.590+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=ECZY6V7E7Q5S type=del 2023-12-03T11:30:45.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.593+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.0 size=1048576 2023-12-03T11:30:45.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.594+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.1 size=1048576 2023-12-03T11:30:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.595+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.2 size=887625 2023-12-03T11:30:45.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.595+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=3CDKAG6CAARI tag=2DST2KRBGV34 type=write size=4033353 2023-12-03T11:30:45.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.596+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~JCY5AP1RUPTJ.0 size=609884 2023-12-03T11:30:45.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.598+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=JCY5AP1RUPTJ tag=BLTS7NIM6BK5 type=write size=1658460 2023-12-03T11:30:45.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.599+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=OC4AGTN7JDYS type=del size=1586230 2023-12-03T11:30:45.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.601+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.0 size=1048576 2023-12-03T11:30:45.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.602+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.1 size=1048576 2023-12-03T11:30:45.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.603+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~IFO0ELQSQG97.2 size=755740 2023-12-03T11:30:45.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.605+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=IFO0ELQSQG97 tag=ZHILJ316GA4O type=write size=3901468 2023-12-03T11:30:45.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.606+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=N9D5RAJA5NL3 type=del size=1979740 2023-12-03T11:30:45.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.607+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=MNUJ26OTYF24 type=del 2023-12-03T11:30:45.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.608+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=DSPBUKSVHMXV type=del size=2093227 2023-12-03T11:30:45.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.610+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=I32ZH4F2UDM9 type=del 2023-12-03T11:30:45.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.611+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=R2CN3XQ1THI9 type=del size=2090948 2023-12-03T11:30:45.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.614+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=DJX4X9RXPTU6 type=del 2023-12-03T11:30:45.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.616+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.0 size=1048576 2023-12-03T11:30:45.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.617+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.1 size=1048576 2023-12-03T11:30:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.618+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.2 size=98877 2023-12-03T11:30:45.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.618+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=HRHBM71UB6Z0 tag=1NGMUY4NS9WG type=write size=3244605 2023-12-03T11:30:45.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.619+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=8N6KBRVCKM5T type=del size=3525965 2023-12-03T11:30:45.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.620+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=HZ9REUU9G016 type=del size=4066370 2023-12-03T11:30:45.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.623+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=IMBLW657FPRJ type=del 2023-12-03T11:30:45.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.625+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.0 size=1048576 2023-12-03T11:30:45.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.626+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.1 size=1048576 2023-12-03T11:30:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.627+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.2 size=59115 2023-12-03T11:30:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.627+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=IQX0C97405EO tag=0L9LDHAP31XE type=write size=3204843 2023-12-03T11:30:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.628+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~FFFAK1T25KAT.0 size=1048576 2023-12-03T11:30:45.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.628+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~FFFAK1T25KAT.1 size=746689 2023-12-03T11:30:45.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.630+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=FFFAK1T25KAT tag=HNOZDF00QG80 type=write size=2843841 2023-12-03T11:30:45.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.632+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~M072T94PIM5T.0 size=128640 2023-12-03T11:30:45.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.632+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=M072T94PIM5T tag=55F4K2PB4KYP type=write size=1177216 2023-12-03T11:30:45.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.634+0000 7fc553ded780 1 < canceled operation key=PL22GPUFN3O0 tag=O9XYOEUFF2EB type=del size=3465451 2023-12-03T11:30:45.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.635+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=GKE0ZD3QDO8E type=del size=236743 2023-12-03T11:30:45.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.636+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=H34BNK6O9B2T type=del size=4146603: (22) Invalid argument 2023-12-03T11:30:45.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.639+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~2D5PUEF8Q3U2.0 size=324893 2023-12-03T11:30:45.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.639+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=2D5PUEF8Q3U2 tag=WWYH6C4K0CA0 type=write size=1373469 2023-12-03T11:30:45.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.640+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=8K34VW59K66J type=del 2023-12-03T11:30:45.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.642+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=297GUAL5SIMZ type=del size=1562546 2023-12-03T11:30:45.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.643+0000 7fc553ded780 1 < canceled operation key=JUDXD9DL4N43 tag=H7WU6WWI0XED type=del size=2882121 2023-12-03T11:30:45.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.645+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.0 size=1048576 2023-12-03T11:30:45.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.646+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.1 size=1048576 2023-12-03T11:30:45.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.647+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.2 size=12182 2023-12-03T11:30:45.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.647+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=J3FHNGOC23AD tag=MQQPWB9W8UFO type=write size=3157910 2023-12-03T11:30:45.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.649+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~KDQZY2FZ8V6E.0 size=924358 2023-12-03T11:30:45.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.651+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=KDQZY2FZ8V6E tag=OS3UOS9TRM1Y type=write size=1972934 2023-12-03T11:30:45.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.653+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~46SUB02DSKGX.0 size=998332 2023-12-03T11:30:45.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.653+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=46SUB02DSKGX tag=M5QVUQLKX5GO type=write size=2046908 2023-12-03T11:30:45.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.656+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=HAMC8JZAQI8Z type=del size=3610810 2023-12-03T11:30:45.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.657+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=FCOT0646VM3Z type=del size=2769888 2023-12-03T11:30:45.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.659+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~K12F09SRBQPQ.0 size=305376 2023-12-03T11:30:45.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.661+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=K12F09SRBQPQ tag=0G5EUCZB7VJN type=write size=1353952 2023-12-03T11:30:45.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.663+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~A216F3FAD0Y8.0 size=313125 2023-12-03T11:30:45.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.663+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=A216F3FAD0Y8 tag=DFDK0CN37QHH type=write size=1361701 2023-12-03T11:30:45.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.665+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.0 size=1048576 2023-12-03T11:30:45.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.666+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.1 size=1048576 2023-12-03T11:30:45.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.667+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~3CDKAG6CAARI.2 size=887625 2023-12-03T11:30:45.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.669+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=3CDKAG6CAARI tag=2DST2KRBGV34 type=write size=4033353 2023-12-03T11:30:45.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.671+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=MAVRMAH0AML6 type=del size=3141329 2023-12-03T11:30:45.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.672+0000 7fc553ded780 1 < canceled operation key=L4ME9RYRR4NG tag=OC4AGTN7JDYS type=del size=1586230 2023-12-03T11:30:45.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.674+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=ROR5J04682HS type=del size=3739406 2023-12-03T11:30:45.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.675+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=N9D5RAJA5NL3 type=del 2023-12-03T11:30:45.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.676+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=ZP3LO5I2VOQP type=del size=1345138 2023-12-03T11:30:45.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.679+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=DSPBUKSVHMXV type=del 2023-12-03T11:30:45.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.681+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~8HRZJI41Y002.0 size=1048576 2023-12-03T11:30:45.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.682+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~8HRZJI41Y002.1 size=862560 2023-12-03T11:30:45.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.682+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=8HRZJI41Y002 tag=Y5BYL5D13OYY type=write size=2959712 2023-12-03T11:30:45.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.684+0000 7fc553ded780 -1 < failed to complete operation key=WOKG1DISIKK0 tag=R2CN3XQ1THI9 type=del size=2090948: (22) Invalid argument 2023-12-03T11:30:45.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.685+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=ERASPSY3KK5V type=write size=259043 2023-12-03T11:30:45.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.686+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.0 size=1048576 2023-12-03T11:30:45.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.687+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.1 size=1048576 2023-12-03T11:30:45.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.689+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~HRHBM71UB6Z0.2 size=98877 2023-12-03T11:30:45.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.691+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=HRHBM71UB6Z0 tag=1NGMUY4NS9WG type=write size=3244605 2023-12-03T11:30:45.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.692+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=WAYEX4QQMHI2 type=del size=1888329 2023-12-03T11:30:45.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.693+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=HZ9REUU9G016 type=del 2023-12-03T11:30:45.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.694+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=BXPBQ3675EFZ type=del size=1535951 2023-12-03T11:30:45.697 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.696+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.0 size=1048576 2023-12-03T11:30:45.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.697+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.1 size=1048576 2023-12-03T11:30:45.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.698+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~IQX0C97405EO.2 size=59115 2023-12-03T11:30:45.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.700+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=IQX0C97405EO tag=0L9LDHAP31XE type=write size=3204843 2023-12-03T11:30:45.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.702+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.0 size=1048576 2023-12-03T11:30:45.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.703+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.1 size=1048576 2023-12-03T11:30:45.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.704+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.2 size=101285 2023-12-03T11:30:45.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.704+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=JS89TQRZRL6P tag=LQAIT2R12V1V type=write size=3247013 2023-12-03T11:30:45.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.706+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~M072T94PIM5T.0 size=128640 2023-12-03T11:30:45.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.708+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=M072T94PIM5T tag=55F4K2PB4KYP type=write size=1177216 2023-12-03T11:30:45.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.710+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~DSFGIYBMPB1H.0 size=1048576 2023-12-03T11:30:45.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.711+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~DSFGIYBMPB1H.1 size=114342 2023-12-03T11:30:45.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.711+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=DSFGIYBMPB1H tag=RWVCB54SC6YZ type=write size=2211494 2023-12-03T11:30:45.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.712+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=GKE0ZD3QDO8E type=del 2023-12-03T11:30:45.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.713+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=XMH54R8L485E type=del size=1938644 2023-12-03T11:30:45.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.715+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~2D5PUEF8Q3U2.0 size=324893 2023-12-03T11:30:45.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.717+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=2D5PUEF8Q3U2 tag=WWYH6C4K0CA0 type=write size=1373469 2023-12-03T11:30:45.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.719+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~NZ3ZQFZZPWBN.0 size=1048576 2023-12-03T11:30:45.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.720+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~NZ3ZQFZZPWBN.1 size=263946 2023-12-03T11:30:45.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.720+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=NZ3ZQFZZPWBN tag=KUO2Z5DKOU38 type=write size=2361098 2023-12-03T11:30:45.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.721+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=297GUAL5SIMZ type=del 2023-12-03T11:30:45.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.723+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=0MIUR1VJU0X9 type=del size=3655194 2023-12-03T11:30:45.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.724+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.0 size=1048576 2023-12-03T11:30:45.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.725+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.1 size=1048576 2023-12-03T11:30:45.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.726+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~J3FHNGOC23AD.2 size=12182 2023-12-03T11:30:45.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.728+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=J3FHNGOC23AD tag=MQQPWB9W8UFO type=write size=3157910 2023-12-03T11:30:45.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.730+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=9L0KN6MXHIAZ type=del size=1156570 2023-12-03T11:30:45.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.731+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~46SUB02DSKGX.0 size=998332 2023-12-03T11:30:45.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.733+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=46SUB02DSKGX tag=M5QVUQLKX5GO type=write size=2046908 2023-12-03T11:30:45.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.735+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=RF44T5QH3BWM type=del size=3284075 2023-12-03T11:30:45.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.736+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=FCOT0646VM3Z type=del 2023-12-03T11:30:45.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.737+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=WSWV6C3TSFSX type=del size=2200100 2023-12-03T11:30:45.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.739+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~A216F3FAD0Y8.0 size=313125 2023-12-03T11:30:45.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.741+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=A216F3FAD0Y8 tag=DFDK0CN37QHH type=write size=1361701 2023-12-03T11:30:45.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.743+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~UO855CCFN3KC.0 size=1048576 2023-12-03T11:30:45.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.744+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~UO855CCFN3KC.1 size=864068 2023-12-03T11:30:45.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.744+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=UO855CCFN3KC tag=W8PJT7XE3SBQ type=write size=2961220 2023-12-03T11:30:45.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.745+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=MAVRMAH0AML6 type=del 2023-12-03T11:30:45.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.746+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=1VJRXSO21HPG type=del size=3614027 2023-12-03T11:30:45.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.748+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=ROR5J04682HS type=del 2023-12-03T11:30:45.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.749+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=6P3KV94WD2NS type=del size=4061829 2023-12-03T11:30:45.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.750+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=ZP3LO5I2VOQP type=del 2023-12-03T11:30:45.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.751+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=7SYX8TUQG7PK type=del size=1117838 2023-12-03T11:30:45.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.753+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~8HRZJI41Y002.0 size=1048576 2023-12-03T11:30:45.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.754+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~8HRZJI41Y002.1 size=862560 2023-12-03T11:30:45.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.756+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=8HRZJI41Y002 tag=Y5BYL5D13OYY type=write size=2959712 2023-12-03T11:30:45.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.757+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=VPZTGNF4IV7A type=del size=2917869 2023-12-03T11:30:45.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.759+0000 7fc553ded780 1 < completed write operation key=ZUNOFY5N3J3H tag=ERASPSY3KK5V type=write size=259043 2023-12-03T11:30:45.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.761+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~R8CE7U5DH5CD.0 size=446485 2023-12-03T11:30:45.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.761+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=R8CE7U5DH5CD tag=GZN3S9K7ZPR1 type=write size=1495061 2023-12-03T11:30:45.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.763+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=WAYEX4QQMHI2 type=del size=1888329 2023-12-03T11:30:45.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.765+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=K47ZR4Z8KDBZ type=del size=3375723 2023-12-03T11:30:45.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.766+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=BXPBQ3675EFZ type=del 2023-12-03T11:30:45.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.767+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=H9K1WIQX2D8H type=del size=3756419 2023-12-03T11:30:45.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.768+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.0 size=1048576 2023-12-03T11:30:45.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.770+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.1 size=1048576 2023-12-03T11:30:45.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.770+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~JS89TQRZRL6P.2 size=101285 2023-12-03T11:30:45.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.773+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=JS89TQRZRL6P tag=LQAIT2R12V1V type=write size=3247013 2023-12-03T11:30:45.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.775+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~0HVQ07CXZ60L.0 size=535652 2023-12-03T11:30:45.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.775+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=0HVQ07CXZ60L tag=O07EZKH4ICOG type=write size=1584228 2023-12-03T11:30:45.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.777+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~DSFGIYBMPB1H.0 size=1048576 2023-12-03T11:30:45.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.777+0000 7fc553ded780 1 < canceled part key=_multipart_JUYA4LXEQ4US.2~DSFGIYBMPB1H.1 size=114342 2023-12-03T11:30:45.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.779+0000 7fc553ded780 1 < canceled multipart upload key=JUYA4LXEQ4US upload=DSFGIYBMPB1H tag=RWVCB54SC6YZ type=write size=2211494 2023-12-03T11:30:45.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.781+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.0 size=1048576 2023-12-03T11:30:45.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.782+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.1 size=1048576 2023-12-03T11:30:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.783+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.2 size=397749 2023-12-03T11:30:45.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.783+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=CFSOSR4CB1KA tag=ZPOXZ1EN99CH type=write size=3543477 2023-12-03T11:30:45.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.784+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=XMH54R8L485E type=del 2023-12-03T11:30:45.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.785+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=T6E31AVYHXNV type=write size=566134 2023-12-03T11:30:45.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.787+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~NZ3ZQFZZPWBN.0 size=1048576 2023-12-03T11:30:45.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.787+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~NZ3ZQFZZPWBN.1 size=263946 2023-12-03T11:30:45.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.789+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=NZ3ZQFZZPWBN tag=KUO2Z5DKOU38 type=write size=2361098 2023-12-03T11:30:45.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.791+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~QDTRTI4GMRCV.0 size=870459 2023-12-03T11:30:45.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.791+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=QDTRTI4GMRCV tag=T7TMQPHMV6N5 type=write size=1919035 2023-12-03T11:30:45.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.792+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=0MIUR1VJU0X9 type=del 2023-12-03T11:30:45.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.795+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~HRWDF9S6FONN.0 size=463226 2023-12-03T11:30:45.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.795+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=HRWDF9S6FONN tag=TI7LZE15QB8I type=write size=1511802 2023-12-03T11:30:45.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.798+0000 7fc553ded780 1 < suggested update operation key=WOKG1DISIKK0 tag=9L0KN6MXHIAZ type=del size=1156570 2023-12-03T11:30:45.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.801+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=I3G83GFYN2UN type=del size=3928363 2023-12-03T11:30:45.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.802+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=RF44T5QH3BWM type=del 2023-12-03T11:30:45.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.804+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=RSM33C35E0YY type=del size=135238 2023-12-03T11:30:45.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.805+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=WSWV6C3TSFSX type=del 2023-12-03T11:30:45.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.806+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=ZO6TKF32HLBG type=del size=1022868 2023-12-03T11:30:45.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.807+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~UO855CCFN3KC.0 size=1048576 2023-12-03T11:30:45.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.808+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~UO855CCFN3KC.1 size=864068 2023-12-03T11:30:45.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.810+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=UO855CCFN3KC tag=W8PJT7XE3SBQ type=write size=2961220 2023-12-03T11:30:45.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.812+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~8ACL6WJFRKEO.0 size=951363 2023-12-03T11:30:45.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.812+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=8ACL6WJFRKEO tag=WPNJSKDC6RZB type=write size=1999939 2023-12-03T11:30:45.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.814+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=1VJRXSO21HPG type=del 2023-12-03T11:30:45.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.815+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=HG47Y3EOFI1Y type=del size=3985315 2023-12-03T11:30:45.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.816+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=6P3KV94WD2NS type=del 2023-12-03T11:30:45.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.817+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=UBUKVKRF6JKP type=del size=3256923 2023-12-03T11:30:45.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.818+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=7SYX8TUQG7PK type=del 2023-12-03T11:30:45.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.821+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.0 size=1048576 2023-12-03T11:30:45.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.822+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.1 size=1048576 2023-12-03T11:30:45.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.823+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.2 size=64483 2023-12-03T11:30:45.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.823+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=75ZEQC009KOP tag=L5PNR7E547MO type=write size=3210211 2023-12-03T11:30:45.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.824+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=VPZTGNF4IV7A type=del 2023-12-03T11:30:45.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.826+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.0 size=1048576 2023-12-03T11:30:45.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.827+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.1 size=1048576 2023-12-03T11:30:45.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.828+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.2 size=27410 2023-12-03T11:30:45.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.828+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=JQESH1MD8AHX tag=E1S3PFA8NF6Y type=write size=3173138 2023-12-03T11:30:45.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.829+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~R8CE7U5DH5CD.0 size=446485 2023-12-03T11:30:45.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.831+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=R8CE7U5DH5CD tag=GZN3S9K7ZPR1 type=write size=1495061 2023-12-03T11:30:45.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.833+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=6TS6A4O12QLI type=del size=379645 2023-12-03T11:30:45.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.834+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=K47ZR4Z8KDBZ type=del 2023-12-03T11:30:45.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.836+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.0 size=1048576 2023-12-03T11:30:45.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.837+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.1 size=1048576 2023-12-03T11:30:45.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.838+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.2 size=332133 2023-12-03T11:30:45.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.838+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=PT2OGNGI36RN tag=DV8TCE6L39VF type=write size=3477861 2023-12-03T11:30:45.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.839+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=H9K1WIQX2D8H type=del 2023-12-03T11:30:45.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.841+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=7IZ6QPYM9MH8 type=del size=3989936 2023-12-03T11:30:45.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.842+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~0HVQ07CXZ60L.0 size=535652 2023-12-03T11:30:45.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.844+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=0HVQ07CXZ60L tag=O07EZKH4ICOG type=write size=1584228 2023-12-03T11:30:45.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.846+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~7YB3ZYMPRB78.0 size=1048576 2023-12-03T11:30:45.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.849+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~7YB3ZYMPRB78.1 size=120870 2023-12-03T11:30:45.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.849+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=7YB3ZYMPRB78 tag=XBC5ER05K00B type=write size=2218022 2023-12-03T11:30:45.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.850+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.0 size=1048576 2023-12-03T11:30:45.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.851+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.1 size=1048576 2023-12-03T11:30:45.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.852+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~CFSOSR4CB1KA.2 size=397749 2023-12-03T11:30:45.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.854+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=CFSOSR4CB1KA tag=ZPOXZ1EN99CH type=write size=3543477 2023-12-03T11:30:45.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.856+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=1UYV4LW292Y9 type=del size=2453650 2023-12-03T11:30:45.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.857+0000 7fc553ded780 1 < completed write operation key=SWTRHXTAR0A1 tag=T6E31AVYHXNV type=write size=566134 2023-12-03T11:30:45.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.858+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=46BQOL9Q1M1R type=del size=692302 2023-12-03T11:30:45.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.859+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~QDTRTI4GMRCV.0 size=870459 2023-12-03T11:30:45.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.861+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=QDTRTI4GMRCV tag=T7TMQPHMV6N5 type=write size=1919035 2023-12-03T11:30:45.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.864+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.0 size=1048576 2023-12-03T11:30:45.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.865+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.1 size=1048576 2023-12-03T11:30:45.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.866+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.2 size=989533 2023-12-03T11:30:45.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.866+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=CPS93PUXJVZ5 tag=VOJVNBO09GO9 type=write size=4135261 2023-12-03T11:30:45.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.866+0000 7fc553ded780 1 < canceled part key=_multipart_6OW429S51UWN.2~HRWDF9S6FONN.0 size=463226 2023-12-03T11:30:45.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.868+0000 7fc553ded780 1 < canceled multipart upload key=6OW429S51UWN upload=HRWDF9S6FONN tag=TI7LZE15QB8I type=write size=1511802 2023-12-03T11:30:45.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.870+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.0 size=1048576 2023-12-03T11:30:45.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.871+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.1 size=1048576 2023-12-03T11:30:45.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.872+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.2 size=754195 2023-12-03T11:30:45.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.872+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=P1JS7SPDJL97 tag=YHSZVW5L0XA3 type=write size=3899923 2023-12-03T11:30:45.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.873+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=I3G83GFYN2UN type=del 2023-12-03T11:30:45.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.875+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=TG71NL5MS3BO type=write size=711090 2023-12-03T11:30:45.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.876+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=RSM33C35E0YY type=del 2023-12-03T11:30:45.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.878+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~0BQY418K1V5I.0 size=1048576 2023-12-03T11:30:45.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.879+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~0BQY418K1V5I.1 size=1048559 2023-12-03T11:30:45.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.879+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=0BQY418K1V5I tag=9AFITRHNMUE8 type=write size=3145711 2023-12-03T11:30:45.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.880+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=ZO6TKF32HLBG type=del 2023-12-03T11:30:45.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.882+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=U60QGRKVVE0N type=write size=922816 2023-12-03T11:30:45.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.883+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~8ACL6WJFRKEO.0 size=951363 2023-12-03T11:30:45.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.885+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=8ACL6WJFRKEO tag=WPNJSKDC6RZB type=write size=1999939 2023-12-03T11:30:45.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.887+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~81C6B96OKDCM.0 size=790324 2023-12-03T11:30:45.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.887+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=81C6B96OKDCM tag=D3YGFTLRZMUQ type=write size=1838900 2023-12-03T11:30:45.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.890+0000 7fc553ded780 1 < suggested remove operation key=J664UUE697Q8 tag=HG47Y3EOFI1Y type=del 2023-12-03T11:30:45.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.891+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=GII7YYRFVDT3 type=del size=204448 2023-12-03T11:30:45.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.892+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=UBUKVKRF6JKP type=del 2023-12-03T11:30:45.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.893+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=3SCAHBFKBJMO type=del size=1923609 2023-12-03T11:30:45.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.894+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.0 size=1048576 2023-12-03T11:30:45.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.895+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.1 size=1048576 2023-12-03T11:30:45.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.896+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~75ZEQC009KOP.2 size=64483 2023-12-03T11:30:45.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.898+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=75ZEQC009KOP tag=L5PNR7E547MO type=write size=3210211 2023-12-03T11:30:45.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.899+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=2GELQ4G1A87I type=write size=48666 2023-12-03T11:30:45.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.901+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.0 size=1048576 2023-12-03T11:30:45.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.902+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.1 size=1048576 2023-12-03T11:30:45.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.903+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~JQESH1MD8AHX.2 size=27410 2023-12-03T11:30:45.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.905+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=JQESH1MD8AHX tag=E1S3PFA8NF6Y type=write size=3173138 2023-12-03T11:30:45.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.907+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.0 size=1048576 2023-12-03T11:30:45.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.908+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.1 size=1048576 2023-12-03T11:30:45.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.909+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.2 size=979834 2023-12-03T11:30:45.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.909+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=9CQ6PSISWSZ7 tag=IE3YH13ZCW1N type=write size=4125562 2023-12-03T11:30:45.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.911+0000 7fc553ded780 1 < suggested remove operation key=UKAL8FCVIRSH tag=6TS6A4O12QLI type=del 2023-12-03T11:30:45.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.915+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.0 size=1048576 2023-12-03T11:30:45.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.916+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.1 size=1048576 2023-12-03T11:30:45.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.917+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.2 size=496094 2023-12-03T11:30:45.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.917+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=DQ4AQ42M3WWW tag=ZNNZ36PDOFYJ type=write size=3641822 2023-12-03T11:30:45.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.919+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.0 size=1048576 2023-12-03T11:30:45.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.919+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.1 size=1048576 2023-12-03T11:30:45.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.920+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~PT2OGNGI36RN.2 size=332133 2023-12-03T11:30:45.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.922+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=PT2OGNGI36RN tag=DV8TCE6L39VF type=write size=3477861 2023-12-03T11:30:45.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.923+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=SXLLPQEN5637 type=del size=337526 2023-12-03T11:30:45.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.924+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=7IZ6QPYM9MH8 type=del 2023-12-03T11:30:45.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.926+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~TKQK6HMJEN3F.0 size=130418 2023-12-03T11:30:45.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.926+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=TKQK6HMJEN3F tag=UAKVTUR41M63 type=write size=1178994 2023-12-03T11:30:45.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.928+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~7YB3ZYMPRB78.0 size=1048576 2023-12-03T11:30:45.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.929+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~7YB3ZYMPRB78.1 size=120870 2023-12-03T11:30:45.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.931+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=7YB3ZYMPRB78 tag=XBC5ER05K00B type=write size=2218022 2023-12-03T11:30:45.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.933+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~XZY7JKDOAZBS.0 size=270590 2023-12-03T11:30:45.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.933+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=XZY7JKDOAZBS tag=UOL0M3FKY8EP type=write size=1319166 2023-12-03T11:30:45.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.934+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=1UYV4LW292Y9 type=del 2023-12-03T11:30:45.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.936+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=DSZ9I51VVQS9 type=del size=2008727 2023-12-03T11:30:45.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.937+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=46BQOL9Q1M1R type=del 2023-12-03T11:30:45.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.939+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~AGAURWHSNPYE.0 size=1048576 2023-12-03T11:30:45.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.940+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~AGAURWHSNPYE.1 size=771965 2023-12-03T11:30:45.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.940+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=AGAURWHSNPYE tag=YVQ8XODZ8IW7 type=write size=2869117 2023-12-03T11:30:45.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.942+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.0 size=1048576 2023-12-03T11:30:45.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.943+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.1 size=1048576 2023-12-03T11:30:45.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.944+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~CPS93PUXJVZ5.2 size=989533 2023-12-03T11:30:45.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.946+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=CPS93PUXJVZ5 tag=VOJVNBO09GO9 type=write size=4135261 2023-12-03T11:30:45.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.948+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=IRQB2VECGF31 type=del size=363002 2023-12-03T11:30:45.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.949+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.0 size=1048576 2023-12-03T11:30:45.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.950+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.1 size=1048576 2023-12-03T11:30:45.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.951+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~P1JS7SPDJL97.2 size=754195 2023-12-03T11:30:45.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.955+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=P1JS7SPDJL97 tag=YHSZVW5L0XA3 type=write size=3899923 2023-12-03T11:30:45.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.956+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=QIQVUP346KRY type=del size=674063 2023-12-03T11:30:45.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.958+0000 7fc553ded780 1 < completed write operation key=FC4WU6SMNEWV tag=TG71NL5MS3BO type=write size=711090 2023-12-03T11:30:45.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.959+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=Q13EDOU4QBKM type=write size=356382 2023-12-03T11:30:45.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.960+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~0BQY418K1V5I.0 size=1048576 2023-12-03T11:30:45.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.962+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~0BQY418K1V5I.1 size=1048559 2023-12-03T11:30:45.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.964+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=0BQY418K1V5I tag=9AFITRHNMUE8 type=write size=3145711 2023-12-03T11:30:45.967 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.966+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~GL8WZI4AUUIQ.0 size=1048576 2023-12-03T11:30:45.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.968+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~GL8WZI4AUUIQ.1 size=329641 2023-12-03T11:30:45.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.968+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=GL8WZI4AUUIQ tag=SUVSHJK09UA8 type=write size=2426793 2023-12-03T11:30:45.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.969+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=U60QGRKVVE0N type=write size=922816 2023-12-03T11:30:45.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.972+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~IFIZGCQU6S62.0 size=1048576 2023-12-03T11:30:45.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.973+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~IFIZGCQU6S62.1 size=632984 2023-12-03T11:30:45.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.973+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=IFIZGCQU6S62 tag=UH71BSJ2YOQ5 type=write size=2730136 2023-12-03T11:30:45.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.975+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~81C6B96OKDCM.0 size=790324 2023-12-03T11:30:45.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.977+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=81C6B96OKDCM tag=D3YGFTLRZMUQ type=write size=1838900 2023-12-03T11:30:45.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.980+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~XWB3S1R29JOE.0 size=1048576 2023-12-03T11:30:45.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.981+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~XWB3S1R29JOE.1 size=706684 2023-12-03T11:30:45.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.981+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=XWB3S1R29JOE tag=KZSOTSI2D4VI type=write size=2803836 2023-12-03T11:30:45.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.983+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=GII7YYRFVDT3 type=del 2023-12-03T11:30:45.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.985+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~O326SNU74CIJ.0 size=814592 2023-12-03T11:30:45.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.985+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=O326SNU74CIJ tag=FIEKI50ULJ0M type=write size=1863168 2023-12-03T11:30:45.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.987+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=3SCAHBFKBJMO type=del 2023-12-03T11:30:45.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.989+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~DYZEJJT2TJVG.0 size=1048576 2023-12-03T11:30:45.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.990+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~DYZEJJT2TJVG.1 size=160068 2023-12-03T11:30:45.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.990+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=DYZEJJT2TJVG tag=0PB4L5B0VZJQ type=write size=2257220 2023-12-03T11:30:45.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.993+0000 7fc553ded780 1 < suggested remove operation key=JUDXD9DL4N43 tag=2GELQ4G1A87I type=write 2023-12-03T11:30:45.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.997+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.0 size=1048576 2023-12-03T11:30:45.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.997+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.1 size=1048576 2023-12-03T11:30:45.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.998+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.2 size=980739 2023-12-03T11:30:45.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.998+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=GSIHAUQ54Y6R tag=1BS4Z2SS95CS type=write size=4126467 2023-12-03T11:30:46.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:45.999+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.0 size=1048576 2023-12-03T11:30:46.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.000+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.1 size=1048576 2023-12-03T11:30:46.002 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.001+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9CQ6PSISWSZ7.2 size=979834 2023-12-03T11:30:46.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.004+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=9CQ6PSISWSZ7 tag=IE3YH13ZCW1N type=write size=4125562 2023-12-03T11:30:46.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.006+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~OOB8J2YOSRIU.0 size=1048576 2023-12-03T11:30:46.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.007+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~OOB8J2YOSRIU.1 size=291618 2023-12-03T11:30:46.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.007+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=OOB8J2YOSRIU tag=L5B3NQ05NNZE type=write size=2388770 2023-12-03T11:30:46.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.009+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.0 size=1048576 2023-12-03T11:30:46.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.010+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.1 size=1048576 2023-12-03T11:30:46.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.011+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~DQ4AQ42M3WWW.2 size=496094 2023-12-03T11:30:46.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.013+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=DQ4AQ42M3WWW tag=ZNNZ36PDOFYJ type=write size=3641822 2023-12-03T11:30:46.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.015+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~44W3TGKF8WWR.0 size=1048576 2023-12-03T11:30:46.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.016+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~44W3TGKF8WWR.1 size=184062 2023-12-03T11:30:46.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.016+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=44W3TGKF8WWR tag=MFXMS9NYATBE type=write size=2281214 2023-12-03T11:30:46.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.018+0000 7fc553ded780 1 < suggested update operation key=YE73HBF64CS7 tag=SXLLPQEN5637 type=del size=337526 2023-12-03T11:30:46.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.021+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~JMDEG3234PY9.0 size=95464 2023-12-03T11:30:46.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.021+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=JMDEG3234PY9 tag=U1TEPW566FCR type=write size=1144040 2023-12-03T11:30:46.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.022+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~TKQK6HMJEN3F.0 size=130418 2023-12-03T11:30:46.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.024+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=TKQK6HMJEN3F tag=UAKVTUR41M63 type=write size=1178994 2023-12-03T11:30:46.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.025+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=VL25EE4QV25X type=del size=1274525 2023-12-03T11:30:46.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.026+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~XZY7JKDOAZBS.0 size=270590 2023-12-03T11:30:46.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.028+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=XZY7JKDOAZBS tag=UOL0M3FKY8EP type=write size=1319166 2023-12-03T11:30:46.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.029+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=X0I1VCRI7TMJ type=del size=1273373 2023-12-03T11:30:46.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.031+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=DSZ9I51VVQS9 type=del 2023-12-03T11:30:46.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.033+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.0 size=1048576 2023-12-03T11:30:46.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.034+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.1 size=1048576 2023-12-03T11:30:46.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.035+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.2 size=13375 2023-12-03T11:30:46.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.035+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=DL243ZG2840Q tag=WTITH3OLYTEY type=write size=3159103 2023-12-03T11:30:46.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.038+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~AGAURWHSNPYE.0 size=1048576 2023-12-03T11:30:46.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.039+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~AGAURWHSNPYE.1 size=771965 2023-12-03T11:30:46.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.041+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=AGAURWHSNPYE tag=YVQ8XODZ8IW7 type=write size=2869117 2023-12-03T11:30:46.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.042+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=PSGOLDN899R7 type=del size=3873057 2023-12-03T11:30:46.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.043+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=IRQB2VECGF31 type=del 2023-12-03T11:30:46.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.045+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~81K0U5U4NY86.0 size=397223 2023-12-03T11:30:46.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.045+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=81K0U5U4NY86 tag=N7S2PU3NZISI type=write size=1445799 2023-12-03T11:30:46.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.048+0000 7fc553ded780 1 < suggested update operation key=C2TPF9Q5Z7XQ tag=QIQVUP346KRY type=del size=674063 2023-12-03T11:30:46.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.049+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=E8VHKCH9O0ED type=del size=2660723 2023-12-03T11:30:46.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.050+0000 7fc553ded780 -1 < failed to complete operation key=JUDXD9DL4N43 tag=Q13EDOU4QBKM type=write size=356382: (22) Invalid argument 2023-12-03T11:30:46.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.051+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=DPO5Q1O3B55S type=del size=723940 2023-12-03T11:30:46.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.053+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~GL8WZI4AUUIQ.0 size=1048576 2023-12-03T11:30:46.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.054+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~GL8WZI4AUUIQ.1 size=329641 2023-12-03T11:30:46.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.056+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=GL8WZI4AUUIQ tag=SUVSHJK09UA8 type=write size=2426793 2023-12-03T11:30:46.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.057+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=SCS30QYMIK0H type=del size=3552757 2023-12-03T11:30:46.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.058+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~IFIZGCQU6S62.0 size=1048576 2023-12-03T11:30:46.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.059+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~IFIZGCQU6S62.1 size=632984 2023-12-03T11:30:46.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.061+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=IFIZGCQU6S62 tag=UH71BSJ2YOQ5 type=write size=2730136 2023-12-03T11:30:46.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.063+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=5PFLX9FMST1R type=del size=1185695 2023-12-03T11:30:46.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.064+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~XWB3S1R29JOE.0 size=1048576 2023-12-03T11:30:46.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.065+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~XWB3S1R29JOE.1 size=706684 2023-12-03T11:30:46.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.067+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=XWB3S1R29JOE tag=KZSOTSI2D4VI type=write size=2803836 2023-12-03T11:30:46.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.069+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~Q1WQ8A23AQCX.0 size=1048576 2023-12-03T11:30:46.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.070+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~Q1WQ8A23AQCX.1 size=411481 2023-12-03T11:30:46.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.070+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=Q1WQ8A23AQCX tag=0Z7IC2B2HH4H type=write size=2508633 2023-12-03T11:30:46.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.071+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~O326SNU74CIJ.0 size=814592 2023-12-03T11:30:46.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.073+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=O326SNU74CIJ tag=FIEKI50ULJ0M type=write size=1863168 2023-12-03T11:30:46.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.074+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=78V1ABEQB326 type=del size=3386966 2023-12-03T11:30:46.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.076+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~DYZEJJT2TJVG.0 size=1048576 2023-12-03T11:30:46.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.078+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~DYZEJJT2TJVG.1 size=160068 2023-12-03T11:30:46.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.080+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=DYZEJJT2TJVG tag=0PB4L5B0VZJQ type=write size=2257220 2023-12-03T11:30:46.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.081+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=V3JOTZI7BZ86 type=del size=2393511 2023-12-03T11:30:46.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.083+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.0 size=1048576 2023-12-03T11:30:46.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.084+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.1 size=1048576 2023-12-03T11:30:46.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.085+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~GSIHAUQ54Y6R.2 size=980739 2023-12-03T11:30:46.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.087+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=GSIHAUQ54Y6R tag=1BS4Z2SS95CS type=write size=4126467 2023-12-03T11:30:46.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.088+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=5X9O7GCP81K0 type=write size=413940 2023-12-03T11:30:46.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.089+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~OOB8J2YOSRIU.0 size=1048576 2023-12-03T11:30:46.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.090+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~OOB8J2YOSRIU.1 size=291618 2023-12-03T11:30:46.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.092+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=OOB8J2YOSRIU tag=L5B3NQ05NNZE type=write size=2388770 2023-12-03T11:30:46.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.094+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=Y2CRYDYG665V type=del size=1524805 2023-12-03T11:30:46.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.095+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~44W3TGKF8WWR.0 size=1048576 2023-12-03T11:30:46.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.096+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~44W3TGKF8WWR.1 size=184062 2023-12-03T11:30:46.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.098+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=44W3TGKF8WWR tag=MFXMS9NYATBE type=write size=2281214 2023-12-03T11:30:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.100+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~A2UYIZBIZFF7.0 size=938224 2023-12-03T11:30:46.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.100+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=A2UYIZBIZFF7 tag=Y5O0EGIKGIEV type=write size=1986800 2023-12-03T11:30:46.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.101+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~JMDEG3234PY9.0 size=95464 2023-12-03T11:30:46.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.103+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=JMDEG3234PY9 tag=U1TEPW566FCR type=write size=1144040 2023-12-03T11:30:46.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.106+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~0HSZCFZAAU21.0 size=851437 2023-12-03T11:30:46.107 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.106+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=0HSZCFZAAU21 tag=S5QX0A0GGB8J type=write size=1900013 2023-12-03T11:30:46.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.107+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=VL25EE4QV25X type=del 2023-12-03T11:30:46.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.108+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=9P68IK1D4EN2 type=del size=860781 2023-12-03T11:30:46.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.109+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=X0I1VCRI7TMJ type=del 2023-12-03T11:30:46.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.112+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~8HQP8WUBK1VE.0 size=1048576 2023-12-03T11:30:46.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.113+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~8HQP8WUBK1VE.1 size=876463 2023-12-03T11:30:46.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.113+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=8HQP8WUBK1VE tag=19BJE451Z3J5 type=write size=2973615 2023-12-03T11:30:46.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.114+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.0 size=1048576 2023-12-03T11:30:46.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.115+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.1 size=1048576 2023-12-03T11:30:46.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.116+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~DL243ZG2840Q.2 size=13375 2023-12-03T11:30:46.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.118+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=DL243ZG2840Q tag=WTITH3OLYTEY type=write size=3159103 2023-12-03T11:30:46.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.119+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=M6XF1DZGOJ53 type=del size=1722119 2023-12-03T11:30:46.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.122+0000 7fc553ded780 1 < suggested remove operation key=UKAL8FCVIRSH tag=PSGOLDN899R7 type=del 2023-12-03T11:30:46.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.124+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~RA3NGKVZCZG3.0 size=1017969 2023-12-03T11:30:46.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.124+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=RA3NGKVZCZG3 tag=YQUSF8R7CAHH type=write size=2066545 2023-12-03T11:30:46.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.125+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~81K0U5U4NY86.0 size=397223 2023-12-03T11:30:46.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.127+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=81K0U5U4NY86 tag=N7S2PU3NZISI type=write size=1445799 2023-12-03T11:30:46.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.129+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~1QPKRP9KPTE2.0 size=545631 2023-12-03T11:30:46.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.129+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=1QPKRP9KPTE2 tag=BRG6WT1B8IQS type=write size=1594207 2023-12-03T11:30:46.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.131+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=E8VHKCH9O0ED type=del 2023-12-03T11:30:46.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.132+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=SCLTLAT8MRO4 type=del size=1304680 2023-12-03T11:30:46.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.133+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=DPO5Q1O3B55S type=del 2023-12-03T11:30:46.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.135+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.0 size=1048576 2023-12-03T11:30:46.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.137+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.1 size=1048576 2023-12-03T11:30:46.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.138+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.2 size=223805 2023-12-03T11:30:46.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.138+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=ROXRA8WOIY27 tag=XUBCA0IFUZ6X type=write size=3369533 2023-12-03T11:30:46.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.139+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=SCS30QYMIK0H type=del 2023-12-03T11:30:46.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.141+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~UIBVRZJ63UDV.0 size=1048576 2023-12-03T11:30:46.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.142+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~UIBVRZJ63UDV.1 size=237709 2023-12-03T11:30:46.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.142+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=UIBVRZJ63UDV tag=CD6N216GOOIL type=write size=2334861 2023-12-03T11:30:46.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.143+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=5PFLX9FMST1R type=del 2023-12-03T11:30:46.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.146+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~4WJLSPPGOP4V.0 size=1048576 2023-12-03T11:30:46.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.147+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~4WJLSPPGOP4V.1 size=128748 2023-12-03T11:30:46.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.147+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=4WJLSPPGOP4V tag=CLPZWMQR2AD0 type=write size=2225900 2023-12-03T11:30:46.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.148+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~Q1WQ8A23AQCX.0 size=1048576 2023-12-03T11:30:46.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.149+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~Q1WQ8A23AQCX.1 size=411481 2023-12-03T11:30:46.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.152+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=Q1WQ8A23AQCX tag=0Z7IC2B2HH4H type=write size=2508633 2023-12-03T11:30:46.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.155+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=XXSJ5WYJGF0R type=del size=3443441 2023-12-03T11:30:46.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.158+0000 7fc553ded780 1 < suggested remove operation key=LRN5NYUZ5JT9 tag=78V1ABEQB326 type=del 2023-12-03T11:30:46.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.159+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=F304J9BDU0EV type=write size=906865 2023-12-03T11:30:46.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.160+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=V3JOTZI7BZ86 type=del 2023-12-03T11:30:46.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.161+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=9KVGPKDGGJMZ type=del size=1686921 2023-12-03T11:30:46.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.163+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=5X9O7GCP81K0 type=write size=413940 2023-12-03T11:30:46.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.164+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=6JFAYHKYI5PZ type=del size=2026749 2023-12-03T11:30:46.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.165+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=Y2CRYDYG665V type=del 2023-12-03T11:30:46.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.166+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=QRARR20T563N type=del size=395333 2023-12-03T11:30:46.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.167+0000 7fc553ded780 1 < canceled part key=_multipart_WOKG1DISIKK0.2~A2UYIZBIZFF7.0 size=938224 2023-12-03T11:30:46.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.169+0000 7fc553ded780 1 < canceled multipart upload key=WOKG1DISIKK0 upload=A2UYIZBIZFF7 tag=Y5O0EGIKGIEV type=write size=1986800 2023-12-03T11:30:46.171 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.170+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=WSMXIFOC25D1 type=del size=1083936 2023-12-03T11:30:46.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.171+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~0HSZCFZAAU21.0 size=851437 2023-12-03T11:30:46.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.173+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=0HSZCFZAAU21 tag=S5QX0A0GGB8J type=write size=1900013 2023-12-03T11:30:46.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.174+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=3JT7JXDYW2G4 type=del size=1562797 2023-12-03T11:30:46.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.176+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=9P68IK1D4EN2 type=del size=860781 2023-12-03T11:30:46.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.178+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.0 size=1048576 2023-12-03T11:30:46.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.179+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.1 size=1048576 2023-12-03T11:30:46.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.180+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.2 size=548079 2023-12-03T11:30:46.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.180+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=S73QK3I412HG tag=99D6OEAH710K type=write size=3693807 2023-12-03T11:30:46.182 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.181+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~8HQP8WUBK1VE.0 size=1048576 2023-12-03T11:30:46.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.182+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~8HQP8WUBK1VE.1 size=876463 2023-12-03T11:30:46.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.184+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=8HQP8WUBK1VE tag=19BJE451Z3J5 type=write size=2973615 2023-12-03T11:30:46.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.185+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=PD0TNR1MR6WV type=del size=3495302 2023-12-03T11:30:46.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.187+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=M6XF1DZGOJ53 type=del 2023-12-03T11:30:46.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.188+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=8115ZRCB6OXA type=del size=3610076 2023-12-03T11:30:46.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.189+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~RA3NGKVZCZG3.0 size=1017969 2023-12-03T11:30:46.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.191+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=RA3NGKVZCZG3 tag=YQUSF8R7CAHH type=write size=2066545 2023-12-03T11:30:46.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.193+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~YVXK8KV7ZD4G.0 size=1048576 2023-12-03T11:30:46.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.195+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~YVXK8KV7ZD4G.1 size=822641 2023-12-03T11:30:46.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.195+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=YVXK8KV7ZD4G tag=QBYYWCGBH70E type=write size=2919793 2023-12-03T11:30:46.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.195+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~1QPKRP9KPTE2.0 size=545631 2023-12-03T11:30:46.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.197+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=1QPKRP9KPTE2 tag=BRG6WT1B8IQS type=write size=1594207 2023-12-03T11:30:46.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.199+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=HT3X8UBFRIEW type=del size=2900583 2023-12-03T11:30:46.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.202+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=SCLTLAT8MRO4 type=del 2023-12-03T11:30:46.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.203+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=NYNC6XLHWILN type=del size=1939438 2023-12-03T11:30:46.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.205+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.0 size=1048576 2023-12-03T11:30:46.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.205+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.1 size=1048576 2023-12-03T11:30:46.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.206+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~ROXRA8WOIY27.2 size=223805 2023-12-03T11:30:46.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.209+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=ROXRA8WOIY27 tag=XUBCA0IFUZ6X type=write size=3369533 2023-12-03T11:30:46.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.210+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=IKDMVC85KE4Z type=del size=3183411 2023-12-03T11:30:46.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.211+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~UIBVRZJ63UDV.0 size=1048576 2023-12-03T11:30:46.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.212+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~UIBVRZJ63UDV.1 size=237709 2023-12-03T11:30:46.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.214+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=UIBVRZJ63UDV tag=CD6N216GOOIL type=write size=2334861 2023-12-03T11:30:46.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.215+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=6YDYN74GE74B type=del size=1089797 2023-12-03T11:30:46.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.217+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~4WJLSPPGOP4V.0 size=1048576 2023-12-03T11:30:46.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.218+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~4WJLSPPGOP4V.1 size=128748 2023-12-03T11:30:46.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.220+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=4WJLSPPGOP4V tag=CLPZWMQR2AD0 type=write size=2225900 2023-12-03T11:30:46.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.221+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=G06TTR3WG30X type=del size=2781325 2023-12-03T11:30:46.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.223+0000 7fc553ded780 1 < suggested remove operation key=EK7V6U9DSLD0 tag=XXSJ5WYJGF0R type=del 2023-12-03T11:30:46.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.224+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=HPHT6XQZBF6P type=del size=1220781 2023-12-03T11:30:46.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.225+0000 7fc553ded780 -1 < no bi entry to suggest for operation key=EK7V6U9DSLD0 tag=F304J9BDU0EV type=write size=906865: (2) No such file or directory 2023-12-03T11:30:46.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.226+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=NSBVQZPCML50 type=del size=439318 2023-12-03T11:30:46.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.227+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=9KVGPKDGGJMZ type=del 2023-12-03T11:30:46.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.229+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=47XFFUITD91Q type=write size=848636 2023-12-03T11:30:46.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.230+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=6JFAYHKYI5PZ type=del 2023-12-03T11:30:46.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.231+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=O7K35OS0G3YR type=del size=1025826 2023-12-03T11:30:46.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.232+0000 7fc553ded780 -1 < failed to complete operation key=EK7V6U9DSLD0 tag=QRARR20T563N type=del size=395333: (22) Invalid argument 2023-12-03T11:30:46.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.234+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=B7NRVEXJD213 type=del size=4142510 2023-12-03T11:30:46.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.235+0000 7fc553ded780 1 < canceled operation key=UKAL8FCVIRSH tag=WSMXIFOC25D1 type=del size=1083936 2023-12-03T11:30:46.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.237+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.0 size=1048576 2023-12-03T11:30:46.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.238+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.1 size=1048576 2023-12-03T11:30:46.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.239+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.2 size=818989 2023-12-03T11:30:46.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.239+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=JZ9279VALJ01 tag=R8XE3XXNUIBP type=write size=3964717 2023-12-03T11:30:46.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.240+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=3JT7JXDYW2G4 type=del 2023-12-03T11:30:46.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.241+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=0TNVR7MA82OA type=del size=3727620 2023-12-03T11:30:46.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.243+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.0 size=1048576 2023-12-03T11:30:46.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.244+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.1 size=1048576 2023-12-03T11:30:46.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.244+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~S73QK3I412HG.2 size=548079 2023-12-03T11:30:46.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.246+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=S73QK3I412HG tag=99D6OEAH710K type=write size=3693807 2023-12-03T11:30:46.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.247+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=Z510L48SZZ3I type=del size=3692668 2023-12-03T11:30:46.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.248+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=PD0TNR1MR6WV type=del 2023-12-03T11:30:46.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.250+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=D1ZWA9CJCTEL type=write size=887911 2023-12-03T11:30:46.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.251+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=8115ZRCB6OXA type=del 2023-12-03T11:30:46.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.253+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=AC26GY3SZ4ED type=write size=376316 2023-12-03T11:30:46.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.254+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~YVXK8KV7ZD4G.0 size=1048576 2023-12-03T11:30:46.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.254+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~YVXK8KV7ZD4G.1 size=822641 2023-12-03T11:30:46.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.256+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=YVXK8KV7ZD4G tag=QBYYWCGBH70E type=write size=2919793 2023-12-03T11:30:46.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.258+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=QU07S3JXUGVE type=write size=673418 2023-12-03T11:30:46.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.260+0000 7fc553ded780 1 < suggested update operation key=ZHK2M78A0SLQ tag=HT3X8UBFRIEW type=del size=2900583 2023-12-03T11:30:46.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.263+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.0 size=1048576 2023-12-03T11:30:46.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.264+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.1 size=1048576 2023-12-03T11:30:46.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.265+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.2 size=365274 2023-12-03T11:30:46.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.265+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=6DW664P8F0KU tag=XIZWRP7J2KGU type=write size=3511002 2023-12-03T11:30:46.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.266+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=NYNC6XLHWILN type=del 2023-12-03T11:30:46.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.269+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~LBXY01J23TMP.0 size=1048576 2023-12-03T11:30:46.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.270+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~LBXY01J23TMP.1 size=687567 2023-12-03T11:30:46.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.270+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=LBXY01J23TMP tag=70I6RPDSMM3L type=write size=2784719 2023-12-03T11:30:46.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.271+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=IKDMVC85KE4Z type=del 2023-12-03T11:30:46.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.272+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=KSTQF30LZ4CO type=del size=2953855 2023-12-03T11:30:46.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.276+0000 7fc553ded780 1 < suggested remove operation key=588S2ASN9I0W tag=6YDYN74GE74B type=del 2023-12-03T11:30:46.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.277+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=FNVYFN646U4K type=del size=3126130 2023-12-03T11:30:46.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.279+0000 7fc553ded780 1 < suggested remove operation key=ICDF1EYNI5IU tag=G06TTR3WG30X type=del 2023-12-03T11:30:46.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.281+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~R810HTPZ607F.0 size=283719 2023-12-03T11:30:46.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.281+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=R810HTPZ607F tag=73NV5ADPPYY7 type=write size=1332295 2023-12-03T11:30:46.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.283+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=HPHT6XQZBF6P type=del 2023-12-03T11:30:46.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.284+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=9IFNFCAZIJGB type=write size=925362 2023-12-03T11:30:46.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.286+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=NSBVQZPCML50 type=del 2023-12-03T11:30:46.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.287+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=US06CPPCE12Q type=del size=302288 2023-12-03T11:30:46.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.288+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=47XFFUITD91Q type=write size=848636 2023-12-03T11:30:46.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.290+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~4KEBQWP6BJ8E.0 size=886718 2023-12-03T11:30:46.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.290+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=4KEBQWP6BJ8E tag=JUE1SKP2SGX4 type=write size=1935294 2023-12-03T11:30:46.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.291+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=O7K35OS0G3YR type=del 2023-12-03T11:30:46.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.292+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=TCR117K3FGYU type=write size=9373 2023-12-03T11:30:46.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.294+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=B7NRVEXJD213 type=del 2023-12-03T11:30:46.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.295+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=7XTYYIB251D9 type=del size=1874755 2023-12-03T11:30:46.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.296+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.0 size=1048576 2023-12-03T11:30:46.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.297+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.1 size=1048576 2023-12-03T11:30:46.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.298+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~JZ9279VALJ01.2 size=818989 2023-12-03T11:30:46.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.300+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=JZ9279VALJ01 tag=R8XE3XXNUIBP type=write size=3964717 2023-12-03T11:30:46.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.301+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=KUDPVYML5LNT type=del size=2884138 2023-12-03T11:30:46.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.302+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=0TNVR7MA82OA type=del 2023-12-03T11:30:46.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.303+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=0LGUAEBM20I1 type=del size=2853995 2023-12-03T11:30:46.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.304+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=Z510L48SZZ3I type=del 2023-12-03T11:30:46.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.307+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.0 size=1048576 2023-12-03T11:30:46.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.307+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.1 size=1048576 2023-12-03T11:30:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.308+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.2 size=255267 2023-12-03T11:30:46.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.308+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=GRHFKBGH42S9 tag=2NK3NKNE8J0Z type=write size=3400995 2023-12-03T11:30:46.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.309+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=D1ZWA9CJCTEL type=write size=887911 2023-12-03T11:30:46.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.311+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=NOX2YUA1Z0F1 type=del size=3406652 2023-12-03T11:30:46.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.314+0000 7fc553ded780 1 < suggested update operation key=ZHK2M78A0SLQ tag=AC26GY3SZ4ED type=write size=376316 2023-12-03T11:30:46.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.316+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=C4XSX3HYNUX7 type=write size=386816 2023-12-03T11:30:46.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.317+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=QU07S3JXUGVE type=write size=673418 2023-12-03T11:30:46.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.318+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=OI3BTHBFVFFU type=del size=3806086 2023-12-03T11:30:46.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.319+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.0 size=1048576 2023-12-03T11:30:46.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.320+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.1 size=1048576 2023-12-03T11:30:46.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.321+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~6DW664P8F0KU.2 size=365274 2023-12-03T11:30:46.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.323+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=6DW664P8F0KU tag=XIZWRP7J2KGU type=write size=3511002 2023-12-03T11:30:46.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.325+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~Y2F8KTA0FNA6.0 size=1048576 2023-12-03T11:30:46.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.326+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~Y2F8KTA0FNA6.1 size=410026 2023-12-03T11:30:46.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.326+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=Y2F8KTA0FNA6 tag=8D2RA6RU934J type=write size=2507178 2023-12-03T11:30:46.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.328+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~LBXY01J23TMP.0 size=1048576 2023-12-03T11:30:46.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.328+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~LBXY01J23TMP.1 size=687567 2023-12-03T11:30:46.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.329+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=LBXY01J23TMP tag=70I6RPDSMM3L type=write size=2784719 2023-12-03T11:30:46.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.332+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.0 size=1048576 2023-12-03T11:30:46.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.333+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.1 size=1048576 2023-12-03T11:30:46.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.333+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.2 size=465630 2023-12-03T11:30:46.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.333+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=OC8OARSE60U7 tag=WQMFIZGZTMJ4 type=write size=3611358 2023-12-03T11:30:46.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.335+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=KSTQF30LZ4CO type=del 2023-12-03T11:30:46.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.336+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=AK38M2K6JKU8 type=del size=2236233 2023-12-03T11:30:46.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.337+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=FNVYFN646U4K type=del 2023-12-03T11:30:46.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.338+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=P54B5NYO7TUW type=del size=2764268 2023-12-03T11:30:46.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.338+0000 7fc553ded780 1 < canceled part key=_multipart_FC4WU6SMNEWV.2~R810HTPZ607F.0 size=283719 2023-12-03T11:30:46.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.340+0000 7fc553ded780 1 < canceled multipart upload key=FC4WU6SMNEWV upload=R810HTPZ607F tag=73NV5ADPPYY7 type=write size=1332295 2023-12-03T11:30:46.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.342+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=DSE10V86FP6E type=del size=1470688 2023-12-03T11:30:46.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.343+0000 7fc553ded780 1 < completed write operation key=CZP8OQRU641T tag=9IFNFCAZIJGB type=write size=925362 2023-12-03T11:30:46.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.344+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=65GXJXPX98AG type=del size=3778980 2023-12-03T11:30:46.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.347+0000 7fc553ded780 1 < suggested update operation key=FP2UZMDKYTD1 tag=US06CPPCE12Q type=del size=302288 2023-12-03T11:30:46.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.348+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=7BKPWC6RVET3 type=del size=2921307 2023-12-03T11:30:46.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.350+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~4KEBQWP6BJ8E.0 size=886718 2023-12-03T11:30:46.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.352+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=4KEBQWP6BJ8E tag=JUE1SKP2SGX4 type=write size=1935294 2023-12-03T11:30:46.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.353+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=8PFMPMM8L6G5 type=del size=3338496 2023-12-03T11:30:46.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.355+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=TCR117K3FGYU type=write size=9373 2023-12-03T11:30:46.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.357+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~LQSEYNQWFJE7.0 size=1048576 2023-12-03T11:30:46.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.358+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~LQSEYNQWFJE7.1 size=107377 2023-12-03T11:30:46.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.358+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=LQSEYNQWFJE7 tag=QK3YHH0I1005 type=write size=2204529 2023-12-03T11:30:46.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.359+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=7XTYYIB251D9 type=del 2023-12-03T11:30:46.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.361+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.0 size=1048576 2023-12-03T11:30:46.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.362+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.1 size=1048576 2023-12-03T11:30:46.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.363+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.2 size=11035 2023-12-03T11:30:46.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.363+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=R8WN9PUTW6B0 tag=3CU2R7XJDBQL type=write size=3156763 2023-12-03T11:30:46.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.365+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=KUDPVYML5LNT type=del 2023-12-03T11:30:46.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.367+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.0 size=1048576 2023-12-03T11:30:46.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.368+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.1 size=1048576 2023-12-03T11:30:46.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.369+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.2 size=354781 2023-12-03T11:30:46.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.369+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=5FYLCPSP5Q8I tag=Y47LKREC4DRR type=write size=3500509 2023-12-03T11:30:46.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.370+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=0LGUAEBM20I1 type=del 2023-12-03T11:30:46.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.372+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=QLXYXLIICC7T type=write size=888189 2023-12-03T11:30:46.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.373+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.0 size=1048576 2023-12-03T11:30:46.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.373+0000 7fc553ded780 1 < canceled part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.1 size=1048576 2023-12-03T11:30:46.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.374+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~GRHFKBGH42S9.2 size=255267 2023-12-03T11:30:46.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.376+0000 7fc553ded780 1 < canceled multipart upload key=6OW429S51UWN upload=GRHFKBGH42S9 tag=2NK3NKNE8J0Z type=write size=3400995 2023-12-03T11:30:46.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.377+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=LG5N0MND7TNJ type=del size=1854939 2023-12-03T11:30:46.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.379+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=NOX2YUA1Z0F1 type=del 2023-12-03T11:30:46.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.380+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=PP4WHZ92O8AE type=del size=1529827 2023-12-03T11:30:46.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.383+0000 7fc553ded780 1 < suggested update operation key=CZP8OQRU641T tag=C4XSX3HYNUX7 type=write size=386816 2023-12-03T11:30:46.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.384+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=LJNMW2V53AM2 type=write size=435517 2023-12-03T11:30:46.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.385+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=OI3BTHBFVFFU type=del 2023-12-03T11:30:46.387 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.386+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=ME6KNRWWP6FK type=write size=818200 2023-12-03T11:30:46.388 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.387+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~Y2F8KTA0FNA6.0 size=1048576 2023-12-03T11:30:46.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.389+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~Y2F8KTA0FNA6.1 size=410026 2023-12-03T11:30:46.393 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.392+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=Y2F8KTA0FNA6 tag=8D2RA6RU934J type=write size=2507178 2023-12-03T11:30:46.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.393+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=GW0IY0L1ED4S type=del size=2364866 2023-12-03T11:30:46.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.395+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.0 size=1048576 2023-12-03T11:30:46.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.396+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.1 size=1048576 2023-12-03T11:30:46.398 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.397+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~OC8OARSE60U7.2 size=465630 2023-12-03T11:30:46.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.399+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=OC8OARSE60U7 tag=WQMFIZGZTMJ4 type=write size=3611358 2023-12-03T11:30:46.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.400+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=XXJY2Y01W9K6 type=del size=4041821 2023-12-03T11:30:46.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.401+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=AK38M2K6JKU8 type=del 2023-12-03T11:30:46.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.403+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~QJ3JLVQ0KWCX.0 size=2815 2023-12-03T11:30:46.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.403+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=QJ3JLVQ0KWCX tag=N15J3PXOK16J type=write size=1051391 2023-12-03T11:30:46.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.405+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=P54B5NYO7TUW type=del 2023-12-03T11:30:46.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.406+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=94XIS5ROT870 type=del size=4009866 2023-12-03T11:30:46.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.407+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=DSE10V86FP6E type=del 2023-12-03T11:30:46.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.410+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~FP1C6NQMV4J3.0 size=171077 2023-12-03T11:30:46.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.410+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=FP1C6NQMV4J3 tag=CE0O39J5ZS2P type=write size=1219653 2023-12-03T11:30:46.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.411+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=65GXJXPX98AG type=del 2023-12-03T11:30:46.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.413+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.0 size=1048576 2023-12-03T11:30:46.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.416+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.1 size=1048576 2023-12-03T11:30:46.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.417+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.2 size=560525 2023-12-03T11:30:46.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.417+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=7A07ROPYJR78 tag=E4JK38Q7YX76 type=write size=3706253 2023-12-03T11:30:46.419 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.418+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=7BKPWC6RVET3 type=del 2023-12-03T11:30:46.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.420+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.0 size=1048576 2023-12-03T11:30:46.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.421+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.1 size=1048576 2023-12-03T11:30:46.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.422+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.2 size=634631 2023-12-03T11:30:46.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.422+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=NJI7D62KE7NU tag=E7JD8E1C3LLW type=write size=3780359 2023-12-03T11:30:46.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.423+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=8PFMPMM8L6G5 type=del 2023-12-03T11:30:46.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.424+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=ZW7F3CXARNPY type=del size=1476740 2023-12-03T11:30:46.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.425+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~LQSEYNQWFJE7.0 size=1048576 2023-12-03T11:30:46.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.427+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~LQSEYNQWFJE7.1 size=107377 2023-12-03T11:30:46.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.429+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=LQSEYNQWFJE7 tag=QK3YHH0I1005 type=write size=2204529 2023-12-03T11:30:46.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.430+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=ZSOY1ZD148OJ type=del size=3173244 2023-12-03T11:30:46.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.431+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.0 size=1048576 2023-12-03T11:30:46.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.432+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.1 size=1048576 2023-12-03T11:30:46.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.433+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~R8WN9PUTW6B0.2 size=11035 2023-12-03T11:30:46.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.437+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=R8WN9PUTW6B0 tag=3CU2R7XJDBQL type=write size=3156763 2023-12-03T11:30:46.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.438+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=R28N6KLONRHQ type=del size=47127 2023-12-03T11:30:46.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.440+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.0 size=1048576 2023-12-03T11:30:46.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.441+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.1 size=1048576 2023-12-03T11:30:46.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.442+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~5FYLCPSP5Q8I.2 size=354781 2023-12-03T11:30:46.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.444+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=5FYLCPSP5Q8I tag=Y47LKREC4DRR type=write size=3500509 2023-12-03T11:30:46.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.445+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=5JQQCOMQTWR8 type=write size=413432 2023-12-03T11:30:46.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.446+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=QLXYXLIICC7T type=write size=888189 2023-12-03T11:30:46.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.448+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~BUO5Z0XSMUHU.0 size=1048576 2023-12-03T11:30:46.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.449+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~BUO5Z0XSMUHU.1 size=757852 2023-12-03T11:30:46.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.449+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=BUO5Z0XSMUHU tag=Y3IV4Y6K14ED type=write size=2855004 2023-12-03T11:30:46.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.451+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=LG5N0MND7TNJ type=del 2023-12-03T11:30:46.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.452+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=C0F0I77HL4VU type=write size=590421 2023-12-03T11:30:46.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.454+0000 7fc553ded780 1 < suggested remove operation key=EK7V6U9DSLD0 tag=PP4WHZ92O8AE type=del 2023-12-03T11:30:46.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.455+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=HC0PEPYQUUWW type=del size=3088717 2023-12-03T11:30:46.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.457+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=LJNMW2V53AM2 type=write size=435517 2023-12-03T11:30:46.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.458+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=VCNLU75FN1AM type=del size=2320452 2023-12-03T11:30:46.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.460+0000 7fc553ded780 1 < suggested update operation key=ZHK2M78A0SLQ tag=ME6KNRWWP6FK type=write size=818200 2023-12-03T11:30:46.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.461+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=MLJNQ3IS3KQE type=del size=1575718 2023-12-03T11:30:46.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.462+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=GW0IY0L1ED4S type=del 2023-12-03T11:30:46.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.464+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=7V434T3V8UTQ type=write size=601039 2023-12-03T11:30:46.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.465+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=XXJY2Y01W9K6 type=del 2023-12-03T11:30:46.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.467+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~7YEU7G47VHKA.0 size=395618 2023-12-03T11:30:46.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.467+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=7YEU7G47VHKA tag=NJ4WE6LNQZLU type=write size=1444194 2023-12-03T11:30:46.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.469+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~QJ3JLVQ0KWCX.0 size=2815 2023-12-03T11:30:46.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.470+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=QJ3JLVQ0KWCX tag=N15J3PXOK16J type=write size=1051391 2023-12-03T11:30:46.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.473+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~XFB66CYLZB34.0 size=1048576 2023-12-03T11:30:46.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.474+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~XFB66CYLZB34.1 size=13098 2023-12-03T11:30:46.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.474+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=XFB66CYLZB34 tag=9AHBV4TS9XBT type=write size=2110250 2023-12-03T11:30:46.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.475+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=94XIS5ROT870 type=del 2023-12-03T11:30:46.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.476+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=9NR7PLGN1A8D type=del size=950491 2023-12-03T11:30:46.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.477+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~FP1C6NQMV4J3.0 size=171077 2023-12-03T11:30:46.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.480+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=FP1C6NQMV4J3 tag=CE0O39J5ZS2P type=write size=1219653 2023-12-03T11:30:46.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.481+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=CE4F7L5Q5PC2 type=del size=543707 2023-12-03T11:30:46.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.482+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.0 size=1048576 2023-12-03T11:30:46.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.483+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.1 size=1048576 2023-12-03T11:30:46.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.484+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~7A07ROPYJR78.2 size=560525 2023-12-03T11:30:46.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.486+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=7A07ROPYJR78 tag=E4JK38Q7YX76 type=write size=3706253 2023-12-03T11:30:46.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.487+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=T1S2Y21V2KIQ type=write size=904284 2023-12-03T11:30:46.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.489+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.0 size=1048576 2023-12-03T11:30:46.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.490+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.1 size=1048576 2023-12-03T11:30:46.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.491+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~NJI7D62KE7NU.2 size=634631 2023-12-03T11:30:46.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.493+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=NJI7D62KE7NU tag=E7JD8E1C3LLW type=write size=3780359 2023-12-03T11:30:46.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.495+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.0 size=1048576 2023-12-03T11:30:46.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.496+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.1 size=1048576 2023-12-03T11:30:46.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.497+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.2 size=185908 2023-12-03T11:30:46.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.497+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=1Y3AMW9H9QXY tag=GAJKEUTABCNQ type=write size=3331636 2023-12-03T11:30:46.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.499+0000 7fc553ded780 1 < suggested update operation key=RO0KGI2QMA71 tag=ZW7F3CXARNPY type=del size=1476740 2023-12-03T11:30:46.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.500+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=P27LGISE5C4I type=del size=3245585 2023-12-03T11:30:46.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.501+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=ZSOY1ZD148OJ type=del 2023-12-03T11:30:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.504+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~3XTIPRTQB0YU.0 size=206891 2023-12-03T11:30:46.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.504+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=3XTIPRTQB0YU tag=JZYLH1HSY2IS type=write size=1255467 2023-12-03T11:30:46.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.505+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=R28N6KLONRHQ type=del 2023-12-03T11:30:46.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.508+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.0 size=1048576 2023-12-03T11:30:46.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.511+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.1 size=1048576 2023-12-03T11:30:46.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.512+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.2 size=171666 2023-12-03T11:30:46.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.512+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=J438XALEJLU2 tag=K0D2PGQF8IP3 type=write size=3317394 2023-12-03T11:30:46.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.514+0000 7fc553ded780 1 < suggested update operation key=4TWM1QM1OHOF tag=5JQQCOMQTWR8 type=write size=413432 2023-12-03T11:30:46.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.516+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~2PSHZCNWDJTW.0 size=1048576 2023-12-03T11:30:46.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.517+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~2PSHZCNWDJTW.1 size=715090 2023-12-03T11:30:46.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.517+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=2PSHZCNWDJTW tag=P4IFDQR90GO6 type=write size=2812242 2023-12-03T11:30:46.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.519+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~BUO5Z0XSMUHU.0 size=1048576 2023-12-03T11:30:46.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.520+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~BUO5Z0XSMUHU.1 size=757852 2023-12-03T11:30:46.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.523+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=BUO5Z0XSMUHU tag=Y3IV4Y6K14ED type=write size=2855004 2023-12-03T11:30:46.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.525+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.0 size=1048576 2023-12-03T11:30:46.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.526+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.1 size=1048576 2023-12-03T11:30:46.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.527+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.2 size=1006853 2023-12-03T11:30:46.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.527+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=GZQC8AOEAEXY tag=D2DF0Z8F39TY type=write size=4152581 2023-12-03T11:30:46.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.528+0000 7fc553ded780 1 < completed write operation key=ICDF1EYNI5IU tag=C0F0I77HL4VU type=write size=590421 2023-12-03T11:30:46.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.530+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=4MB0JM9MLAUF type=del size=3996585 2023-12-03T11:30:46.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.531+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=HC0PEPYQUUWW type=del 2023-12-03T11:30:46.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.533+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~SL6FIMYR40BV.0 size=1048576 2023-12-03T11:30:46.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.534+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~SL6FIMYR40BV.1 size=74966 2023-12-03T11:30:46.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.534+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=SL6FIMYR40BV tag=SXM1PZAYN206 type=write size=2172118 2023-12-03T11:30:46.536 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.535+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=VCNLU75FN1AM type=del 2023-12-03T11:30:46.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.537+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=KREL8Z4JL2TP type=write size=897195 2023-12-03T11:30:46.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.539+0000 7fc553ded780 1 < suggested update operation key=ICDF1EYNI5IU tag=MLJNQ3IS3KQE type=del size=1575718 2023-12-03T11:30:46.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.540+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=T8OQVIOULSEU type=del size=2655643 2023-12-03T11:30:46.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.541+0000 7fc553ded780 1 < completed write operation key=588S2ASN9I0W tag=7V434T3V8UTQ type=write size=601039 2023-12-03T11:30:46.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.543+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=2X560AP6GN03 type=del size=326050 2023-12-03T11:30:46.545 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.544+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~7YEU7G47VHKA.0 size=395618 2023-12-03T11:30:46.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.546+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=7YEU7G47VHKA tag=NJ4WE6LNQZLU type=write size=1444194 2023-12-03T11:30:46.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.548+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HN4F9N0ZU9D0.0 size=1048576 2023-12-03T11:30:46.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.549+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~HN4F9N0ZU9D0.1 size=286266 2023-12-03T11:30:46.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.549+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=HN4F9N0ZU9D0 tag=7F9HNEL9SQEV type=write size=2383418 2023-12-03T11:30:46.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.550+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~XFB66CYLZB34.0 size=1048576 2023-12-03T11:30:46.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.551+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~XFB66CYLZB34.1 size=13098 2023-12-03T11:30:46.554 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.554+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=XFB66CYLZB34 tag=9AHBV4TS9XBT type=write size=2110250 2023-12-03T11:30:46.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.556+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~BR4ACA816G7B.0 size=136806 2023-12-03T11:30:46.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.556+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=BR4ACA816G7B tag=7EUU4TTXYLEL type=write size=1185382 2023-12-03T11:30:46.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.557+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=9NR7PLGN1A8D type=del 2023-12-03T11:30:46.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.559+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=Y503U9LXRRT3 type=del size=3520701 2023-12-03T11:30:46.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.561+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=CE4F7L5Q5PC2 type=del 2023-12-03T11:30:46.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.562+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=PNUJUAGUKO2H type=del size=703885 2023-12-03T11:30:46.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.563+0000 7fc553ded780 1 < canceled operation key=JUDXD9DL4N43 tag=T1S2Y21V2KIQ type=write size=904284 2023-12-03T11:30:46.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.564+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=V2U9NG84GXQX type=del size=2629888 2023-12-03T11:30:46.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.566+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.0 size=1048576 2023-12-03T11:30:46.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.567+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.1 size=1048576 2023-12-03T11:30:46.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.567+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~1Y3AMW9H9QXY.2 size=185908 2023-12-03T11:30:46.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.569+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=1Y3AMW9H9QXY tag=GAJKEUTABCNQ type=write size=3331636 2023-12-03T11:30:46.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.572+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~W8HC828001BO.0 size=1048576 2023-12-03T11:30:46.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.573+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~W8HC828001BO.1 size=333968 2023-12-03T11:30:46.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.573+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=W8HC828001BO tag=5SL9A62V08AW type=write size=2431120 2023-12-03T11:30:46.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.574+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=P27LGISE5C4I type=del 2023-12-03T11:30:46.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.576+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~B6O9O3OPR4N3.0 size=943183 2023-12-03T11:30:46.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.576+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=B6O9O3OPR4N3 tag=8RVH9YY6NHTG type=write size=1991759 2023-12-03T11:30:46.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.578+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~3XTIPRTQB0YU.0 size=206891 2023-12-03T11:30:46.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.580+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=3XTIPRTQB0YU tag=JZYLH1HSY2IS type=write size=1255467 2023-12-03T11:30:46.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.581+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=8BLTMQEXK6QW type=del size=1292312 2023-12-03T11:30:46.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.582+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.0 size=1048576 2023-12-03T11:30:46.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.584+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.1 size=1048576 2023-12-03T11:30:46.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.585+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~J438XALEJLU2.2 size=171666 2023-12-03T11:30:46.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.587+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=J438XALEJLU2 tag=K0D2PGQF8IP3 type=write size=3317394 2023-12-03T11:30:46.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.590+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.0 size=1048576 2023-12-03T11:30:46.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.591+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.1 size=1048576 2023-12-03T11:30:46.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.592+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.2 size=205883 2023-12-03T11:30:46.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.592+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=OBGEVO4UC6JQ tag=XZ6K127SKR3B type=write size=3351611 2023-12-03T11:30:46.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.592+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~2PSHZCNWDJTW.0 size=1048576 2023-12-03T11:30:46.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.592+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~2PSHZCNWDJTW.1 size=715090 2023-12-03T11:30:46.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.594+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=2PSHZCNWDJTW tag=P4IFDQR90GO6 type=write size=2812242 2023-12-03T11:30:46.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.596+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=61RSE5ARGJF1 type=del size=2037918 2023-12-03T11:30:46.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.597+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.0 size=1048576 2023-12-03T11:30:46.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.598+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.1 size=1048576 2023-12-03T11:30:46.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.599+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~GZQC8AOEAEXY.2 size=1006853 2023-12-03T11:30:46.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.601+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=GZQC8AOEAEXY tag=D2DF0Z8F39TY type=write size=4152581 2023-12-03T11:30:46.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.602+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=HF3AU9MYHXMP type=del size=3341896 2023-12-03T11:30:46.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.604+0000 7fc553ded780 1 < suggested update operation key=UKAL8FCVIRSH tag=4MB0JM9MLAUF type=del size=3996585 2023-12-03T11:30:46.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.607+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.0 size=1048576 2023-12-03T11:30:46.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.608+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.1 size=1048576 2023-12-03T11:30:46.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.609+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.2 size=143939 2023-12-03T11:30:46.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.609+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=Q2F8V3SW8ZR0 tag=CT02C4VTKP57 type=write size=3289667 2023-12-03T11:30:46.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.610+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~SL6FIMYR40BV.0 size=1048576 2023-12-03T11:30:46.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.611+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~SL6FIMYR40BV.1 size=74966 2023-12-03T11:30:46.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.613+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=SL6FIMYR40BV tag=SXM1PZAYN206 type=write size=2172118 2023-12-03T11:30:46.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.614+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=F6D14QSP0ZK2 type=write size=392697 2023-12-03T11:30:46.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.615+0000 7fc553ded780 1 < completed write operation key=G1OBBD75M5ZQ tag=KREL8Z4JL2TP type=write size=897195 2023-12-03T11:30:46.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.618+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~CW2PHEJNMOEF.0 size=1048576 2023-12-03T11:30:46.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.619+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~CW2PHEJNMOEF.1 size=189135 2023-12-03T11:30:46.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.619+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=CW2PHEJNMOEF tag=1JVIK8QLNNT1 type=write size=2286287 2023-12-03T11:30:46.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.620+0000 7fc553ded780 -1 < failed to complete operation key=UKAL8FCVIRSH tag=T8OQVIOULSEU type=del size=2655643: (22) Invalid argument 2023-12-03T11:30:46.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.623+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~M6LJ2HZMQP0O.0 size=282506 2023-12-03T11:30:46.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.623+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=M6LJ2HZMQP0O tag=9UBBMSE53IR1 type=write size=1331082 2023-12-03T11:30:46.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.624+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=2X560AP6GN03 type=del 2023-12-03T11:30:46.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.627+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~W2IOQRMF5K2M.0 size=1048576 2023-12-03T11:30:46.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.628+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~W2IOQRMF5K2M.1 size=889280 2023-12-03T11:30:46.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.629+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=W2IOQRMF5K2M tag=0UV9N164ZFXT type=write size=2986432 2023-12-03T11:30:46.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.630+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HN4F9N0ZU9D0.0 size=1048576 2023-12-03T11:30:46.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.631+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~HN4F9N0ZU9D0.1 size=286266 2023-12-03T11:30:46.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.633+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=HN4F9N0ZU9D0 tag=7F9HNEL9SQEV type=write size=2383418 2023-12-03T11:30:46.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.635+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=85V8TH133BXB type=del size=159408 2023-12-03T11:30:46.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.636+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~BR4ACA816G7B.0 size=136806 2023-12-03T11:30:46.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.638+0000 7fc553ded780 1 < completed multipart upload key=FC4WU6SMNEWV upload=BR4ACA816G7B tag=7EUU4TTXYLEL type=write size=1185382 2023-12-03T11:30:46.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.641+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=ZHNLYIJ3HEH8 type=del size=2999990 2023-12-03T11:30:46.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.642+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=Y503U9LXRRT3 type=del 2023-12-03T11:30:46.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.644+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=G6ZVU2Q6ATV8 type=del size=3734687 2023-12-03T11:30:46.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.645+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=PNUJUAGUKO2H type=del size=703885 2023-12-03T11:30:46.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.647+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~BXXZYR796OOX.0 size=1048576 2023-12-03T11:30:46.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.648+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~BXXZYR796OOX.1 size=175355 2023-12-03T11:30:46.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.648+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=BXXZYR796OOX tag=UUT9I35DM0FW type=write size=2272507 2023-12-03T11:30:46.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.649+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=V2U9NG84GXQX type=del 2023-12-03T11:30:46.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.651+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=BY5U7W85SQXJ type=del size=664117 2023-12-03T11:30:46.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.652+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~W8HC828001BO.0 size=1048576 2023-12-03T11:30:46.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.653+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~W8HC828001BO.1 size=333968 2023-12-03T11:30:46.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.656+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=W8HC828001BO tag=5SL9A62V08AW type=write size=2431120 2023-12-03T11:30:46.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.658+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9H956586E6S4.0 size=556852 2023-12-03T11:30:46.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.658+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=9H956586E6S4 tag=VOMXIGIJ0YDM type=write size=1605428 2023-12-03T11:30:46.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.660+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~B6O9O3OPR4N3.0 size=943183 2023-12-03T11:30:46.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.662+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=B6O9O3OPR4N3 tag=8RVH9YY6NHTG type=write size=1991759 2023-12-03T11:30:46.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.665+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~AVHRLQ58YZ6F.0 size=1048576 2023-12-03T11:30:46.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.666+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~AVHRLQ58YZ6F.1 size=561723 2023-12-03T11:30:46.667 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.666+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=AVHRLQ58YZ6F tag=Z3ZT08WJNCNK type=write size=2658875 2023-12-03T11:30:46.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.668+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=8BLTMQEXK6QW type=del 2023-12-03T11:30:46.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.670+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.0 size=1048576 2023-12-03T11:30:46.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.672+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.1 size=1048576 2023-12-03T11:30:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.673+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.2 size=94088 2023-12-03T11:30:46.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.673+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=KI6GZDMD485O tag=GS64K2TFTNY2 type=write size=3239816 2023-12-03T11:30:46.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.674+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.0 size=1048576 2023-12-03T11:30:46.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.675+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.1 size=1048576 2023-12-03T11:30:46.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.677+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~OBGEVO4UC6JQ.2 size=205883 2023-12-03T11:30:46.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.679+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=OBGEVO4UC6JQ tag=XZ6K127SKR3B type=write size=3351611 2023-12-03T11:30:46.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.681+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~2CBTZDQRQ29F.0 size=816180 2023-12-03T11:30:46.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.681+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=2CBTZDQRQ29F tag=FI0M4UWVVAR5 type=write size=1864756 2023-12-03T11:30:46.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.682+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=61RSE5ARGJF1 type=del 2023-12-03T11:30:46.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.683+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=EX6W62A810IT type=write size=434935 2023-12-03T11:30:46.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.685+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=HF3AU9MYHXMP type=del 2023-12-03T11:30:46.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.687+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.0 size=1048576 2023-12-03T11:30:46.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.688+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.1 size=1048576 2023-12-03T11:30:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.689+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.2 size=321890 2023-12-03T11:30:46.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.689+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=93K7OUOWMTVR tag=QYE9LCTN95SE type=write size=3467618 2023-12-03T11:30:46.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.690+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.0 size=1048576 2023-12-03T11:30:46.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.691+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.1 size=1048576 2023-12-03T11:30:46.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.692+0000 7fc553ded780 1 < completed part key=_multipart_588S2ASN9I0W.2~Q2F8V3SW8ZR0.2 size=143939 2023-12-03T11:30:46.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.694+0000 7fc553ded780 1 < completed multipart upload key=588S2ASN9I0W upload=Q2F8V3SW8ZR0 tag=CT02C4VTKP57 type=write size=3289667 2023-12-03T11:30:46.697 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.696+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~S222536H6RML.0 size=1048576 2023-12-03T11:30:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.697+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~S222536H6RML.1 size=334236 2023-12-03T11:30:46.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.697+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=S222536H6RML tag=AI6PVJ84NMG1 type=write size=2431388 2023-12-03T11:30:46.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.698+0000 7fc553ded780 1 < completed write operation key=TQRZT37AB9WC tag=F6D14QSP0ZK2 type=write size=392697 2023-12-03T11:30:46.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.699+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=OUGFPAVEK0WG type=del size=604011 2023-12-03T11:30:46.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.701+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~CW2PHEJNMOEF.0 size=1048576 2023-12-03T11:30:46.702 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.701+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~CW2PHEJNMOEF.1 size=189135 2023-12-03T11:30:46.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.703+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=CW2PHEJNMOEF tag=1JVIK8QLNNT1 type=write size=2286287 2023-12-03T11:30:46.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.705+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~ZYFU7EZN4OPD.0 size=285186 2023-12-03T11:30:46.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.705+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=ZYFU7EZN4OPD tag=VKGVZRAXGJ2K type=write size=1333762 2023-12-03T11:30:46.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.706+0000 7fc553ded780 1 < canceled part key=_multipart_IRMADF9MXXBZ.2~M6LJ2HZMQP0O.0 size=282506 2023-12-03T11:30:46.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.708+0000 7fc553ded780 1 < canceled multipart upload key=IRMADF9MXXBZ upload=M6LJ2HZMQP0O tag=9UBBMSE53IR1 type=write size=1331082 2023-12-03T11:30:46.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.710+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~CHD8U475LLOT.0 size=1048576 2023-12-03T11:30:46.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.711+0000 7fc553ded780 1 > prepared part key=_multipart_JUDXD9DL4N43.2~CHD8U475LLOT.1 size=204811 2023-12-03T11:30:46.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.711+0000 7fc553ded780 1 > prepared multipart upload key=JUDXD9DL4N43 upload=CHD8U475LLOT tag=Y5CJM7WZCRQC type=write size=2301963 2023-12-03T11:30:46.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.712+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~W2IOQRMF5K2M.0 size=1048576 2023-12-03T11:30:46.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.713+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~W2IOQRMF5K2M.1 size=889280 2023-12-03T11:30:46.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.715+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=W2IOQRMF5K2M tag=0UV9N164ZFXT type=write size=2986432 2023-12-03T11:30:46.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.716+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=IRH8A1AMSPCV type=del size=228459 2023-12-03T11:30:46.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.717+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=85V8TH133BXB type=del 2023-12-03T11:30:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.719+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~BEFK3MI2RVI6.0 size=698944 2023-12-03T11:30:46.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.719+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=BEFK3MI2RVI6 tag=AONLS91Y3EFE type=write size=1747520 2023-12-03T11:30:46.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.720+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=ZHNLYIJ3HEH8 type=del 2023-12-03T11:30:46.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.722+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=G49ILNL8ENJM type=del size=4051551 2023-12-03T11:30:46.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.724+0000 7fc553ded780 1 < suggested remove operation key=X741JIRT7RIG tag=G6ZVU2Q6ATV8 type=del 2023-12-03T11:30:46.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.725+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=34CKGES9V0UN type=del size=248981 2023-12-03T11:30:46.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.726+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~BXXZYR796OOX.0 size=1048576 2023-12-03T11:30:46.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.727+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~BXXZYR796OOX.1 size=175355 2023-12-03T11:30:46.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.729+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=BXXZYR796OOX tag=UUT9I35DM0FW type=write size=2272507 2023-12-03T11:30:46.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.732+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~YRA68IXW14PA.0 size=677194 2023-12-03T11:30:46.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.732+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=YRA68IXW14PA tag=525BKZNO4WL4 type=write size=1725770 2023-12-03T11:30:46.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.733+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=BY5U7W85SQXJ type=del 2023-12-03T11:30:46.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.734+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=BHMDS9C3NTSE type=del size=369912 2023-12-03T11:30:46.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.735+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9H956586E6S4.0 size=556852 2023-12-03T11:30:46.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.738+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=9H956586E6S4 tag=VOMXIGIJ0YDM type=write size=1605428 2023-12-03T11:30:46.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.739+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=8PYRP4OIS5HM type=del size=3148183 2023-12-03T11:30:46.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.740+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~AVHRLQ58YZ6F.0 size=1048576 2023-12-03T11:30:46.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.741+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~AVHRLQ58YZ6F.1 size=561723 2023-12-03T11:30:46.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.743+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=AVHRLQ58YZ6F tag=Z3ZT08WJNCNK type=write size=2658875 2023-12-03T11:30:46.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.749+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.0 size=1048576 2023-12-03T11:30:46.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.752+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.1 size=1048576 2023-12-03T11:30:46.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.753+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.2 size=878693 2023-12-03T11:30:46.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.753+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=0D3O79DQ3XIN tag=SE8PLUXIUJCZ type=write size=4024421 2023-12-03T11:30:46.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.754+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.0 size=1048576 2023-12-03T11:30:46.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.755+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.1 size=1048576 2023-12-03T11:30:46.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.756+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~KI6GZDMD485O.2 size=94088 2023-12-03T11:30:46.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.758+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=KI6GZDMD485O tag=GS64K2TFTNY2 type=write size=3239816 2023-12-03T11:30:46.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.761+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.0 size=1048576 2023-12-03T11:30:46.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.762+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.1 size=1048576 2023-12-03T11:30:46.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.763+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.2 size=714186 2023-12-03T11:30:46.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.763+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=Y8E08FK9RML5 tag=YEGHXGR8Y210 type=write size=3859914 2023-12-03T11:30:46.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.764+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~2CBTZDQRQ29F.0 size=816180 2023-12-03T11:30:46.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.766+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=2CBTZDQRQ29F tag=FI0M4UWVVAR5 type=write size=1864756 2023-12-03T11:30:46.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.768+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~9IUT2ANW20V5.0 size=1048576 2023-12-03T11:30:46.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.769+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~9IUT2ANW20V5.1 size=547611 2023-12-03T11:30:46.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.769+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=9IUT2ANW20V5 tag=NSM9N82RDHXN type=write size=2644763 2023-12-03T11:30:46.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.772+0000 7fc553ded780 1 < suggested update operation key=TQRZT37AB9WC tag=EX6W62A810IT type=write size=434935 2023-12-03T11:30:46.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.774+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~643ZSVYJ7W6I.0 size=623099 2023-12-03T11:30:46.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.774+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=643ZSVYJ7W6I tag=6ABZ06MG79Q7 type=write size=1671675 2023-12-03T11:30:46.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.775+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.0 size=1048576 2023-12-03T11:30:46.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.776+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.1 size=1048576 2023-12-03T11:30:46.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.777+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~93K7OUOWMTVR.2 size=321890 2023-12-03T11:30:46.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.779+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=93K7OUOWMTVR tag=QYE9LCTN95SE type=write size=3467618 2023-12-03T11:30:46.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.781+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~G311CPZERP42.0 size=1048576 2023-12-03T11:30:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.782+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~G311CPZERP42.1 size=647313 2023-12-03T11:30:46.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.782+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=G311CPZERP42 tag=V024JMU0DK74 type=write size=2744465 2023-12-03T11:30:46.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.783+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~S222536H6RML.0 size=1048576 2023-12-03T11:30:46.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.784+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~S222536H6RML.1 size=334236 2023-12-03T11:30:46.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.786+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=S222536H6RML tag=AI6PVJ84NMG1 type=write size=2431388 2023-12-03T11:30:46.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.788+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.0 size=1048576 2023-12-03T11:30:46.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.789+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.1 size=1048576 2023-12-03T11:30:46.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.790+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.2 size=310624 2023-12-03T11:30:46.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.790+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=DC27O827TQ14 tag=3DFVU9EPBBE5 type=write size=3456352 2023-12-03T11:30:46.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.792+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=OUGFPAVEK0WG type=del 2023-12-03T11:30:46.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.793+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=6UBNCT90KSZM type=del size=1688290 2023-12-03T11:30:46.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.794+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~ZYFU7EZN4OPD.0 size=285186 2023-12-03T11:30:46.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.796+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=ZYFU7EZN4OPD tag=VKGVZRAXGJ2K type=write size=1333762 2023-12-03T11:30:46.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.798+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.0 size=1048576 2023-12-03T11:30:46.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.799+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.1 size=1048576 2023-12-03T11:30:46.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.800+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.2 size=619483 2023-12-03T11:30:46.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.800+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=I0QKB9J67A56 tag=W4EYS3Z17UG9 type=write size=3765211 2023-12-03T11:30:46.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.802+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~CHD8U475LLOT.0 size=1048576 2023-12-03T11:30:46.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.802+0000 7fc553ded780 1 < completed part key=_multipart_JUDXD9DL4N43.2~CHD8U475LLOT.1 size=204811 2023-12-03T11:30:46.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.805+0000 7fc553ded780 1 < completed multipart upload key=JUDXD9DL4N43 upload=CHD8U475LLOT tag=Y5CJM7WZCRQC type=write size=2301963 2023-12-03T11:30:46.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.806+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=Q17M5XODZTEU type=del size=1704092 2023-12-03T11:30:46.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.808+0000 7fc553ded780 1 < suggested update operation key=G1OBBD75M5ZQ tag=IRH8A1AMSPCV type=del size=228459 2023-12-03T11:30:46.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.810+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=8E5JIDQJFCUI type=del size=142688 2023-12-03T11:30:46.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.811+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~BEFK3MI2RVI6.0 size=698944 2023-12-03T11:30:46.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.813+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=BEFK3MI2RVI6 tag=AONLS91Y3EFE type=write size=1747520 2023-12-03T11:30:46.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.814+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=9O4GD8UQXAY2 type=del size=387520 2023-12-03T11:30:46.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.815+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=G49ILNL8ENJM type=del 2023-12-03T11:30:46.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.816+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=VNART804203O type=del size=1127479 2023-12-03T11:30:46.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.818+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=34CKGES9V0UN type=del size=248981 2023-12-03T11:30:46.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.819+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=VY3UDOEK5TWN type=del size=1752225 2023-12-03T11:30:46.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.820+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~YRA68IXW14PA.0 size=677194 2023-12-03T11:30:46.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.822+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=YRA68IXW14PA tag=525BKZNO4WL4 type=write size=1725770 2023-12-03T11:30:46.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.825+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.0 size=1048576 2023-12-03T11:30:46.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.826+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.1 size=1048576 2023-12-03T11:30:46.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.827+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.2 size=154028 2023-12-03T11:30:46.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.827+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=AVNDZH6MHY8L tag=BZXSY65GAL21 type=write size=3299756 2023-12-03T11:30:46.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.828+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=BHMDS9C3NTSE type=del 2023-12-03T11:30:46.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.829+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=FW74550PX5RR type=del size=3222695 2023-12-03T11:30:46.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.830+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=8PYRP4OIS5HM type=del 2023-12-03T11:30:46.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.832+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=Z9TP839G1BAY type=del size=3696409 2023-12-03T11:30:46.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.833+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.0 size=1048576 2023-12-03T11:30:46.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.834+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.1 size=1048576 2023-12-03T11:30:46.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.835+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~0D3O79DQ3XIN.2 size=878693 2023-12-03T11:30:46.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.837+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=0D3O79DQ3XIN tag=SE8PLUXIUJCZ type=write size=4024421 2023-12-03T11:30:46.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.839+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~HF26HVKBYK4W.0 size=1048576 2023-12-03T11:30:46.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.840+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~HF26HVKBYK4W.1 size=242855 2023-12-03T11:30:46.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.840+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=HF26HVKBYK4W tag=8W7V5M0LGIPX type=write size=2340007 2023-12-03T11:30:46.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.841+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.0 size=1048576 2023-12-03T11:30:46.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.843+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.1 size=1048576 2023-12-03T11:30:46.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.844+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~Y8E08FK9RML5.2 size=714186 2023-12-03T11:30:46.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.846+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=Y8E08FK9RML5 tag=YEGHXGR8Y210 type=write size=3859914 2023-12-03T11:30:46.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.848+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~ZJ26WIU7X3ZU.0 size=707391 2023-12-03T11:30:46.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.848+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=ZJ26WIU7X3ZU tag=4JVFXD9O76LJ type=write size=1755967 2023-12-03T11:30:46.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.849+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~9IUT2ANW20V5.0 size=1048576 2023-12-03T11:30:46.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.850+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~9IUT2ANW20V5.1 size=547611 2023-12-03T11:30:46.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.852+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=9IUT2ANW20V5 tag=NSM9N82RDHXN type=write size=2644763 2023-12-03T11:30:46.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.854+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.0 size=1048576 2023-12-03T11:30:46.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.855+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.1 size=1048576 2023-12-03T11:30:46.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.856+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.2 size=673325 2023-12-03T11:30:46.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.856+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=MK6LCSBC640M tag=8YGN2KWAXIV2 type=write size=3819053 2023-12-03T11:30:46.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.858+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~643ZSVYJ7W6I.0 size=623099 2023-12-03T11:30:46.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.861+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=643ZSVYJ7W6I tag=6ABZ06MG79Q7 type=write size=1671675 2023-12-03T11:30:46.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.863+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~XE4OL2HO81IY.0 size=642152 2023-12-03T11:30:46.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.863+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=XE4OL2HO81IY tag=J4CK63FKL2HD type=write size=1690728 2023-12-03T11:30:46.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.864+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~G311CPZERP42.0 size=1048576 2023-12-03T11:30:46.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.865+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~G311CPZERP42.1 size=647313 2023-12-03T11:30:46.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.867+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=G311CPZERP42 tag=V024JMU0DK74 type=write size=2744465 2023-12-03T11:30:46.869 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.868+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=7W3SGR5XN8GV type=del size=1195935 2023-12-03T11:30:46.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.870+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.0 size=1048576 2023-12-03T11:30:46.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.871+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.1 size=1048576 2023-12-03T11:30:46.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.871+0000 7fc553ded780 1 < canceled part key=_multipart_SWTRHXTAR0A1.2~DC27O827TQ14.2 size=310624 2023-12-03T11:30:46.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.873+0000 7fc553ded780 1 < canceled multipart upload key=SWTRHXTAR0A1 upload=DC27O827TQ14 tag=3DFVU9EPBBE5 type=write size=3456352 2023-12-03T11:30:46.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.875+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~VXM97O9N55XB.0 size=1048576 2023-12-03T11:30:46.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.876+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~VXM97O9N55XB.1 size=583505 2023-12-03T11:30:46.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.876+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=VXM97O9N55XB tag=QFKVD0NF1JUJ type=write size=2680657 2023-12-03T11:30:46.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.877+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=6UBNCT90KSZM type=del 2023-12-03T11:30:46.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.878+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=NVQLYZUYFFX6 type=del size=967270 2023-12-03T11:30:46.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.880+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.0 size=1048576 2023-12-03T11:30:46.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.881+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.1 size=1048576 2023-12-03T11:30:46.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.882+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~I0QKB9J67A56.2 size=619483 2023-12-03T11:30:46.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.884+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=I0QKB9J67A56 tag=W4EYS3Z17UG9 type=write size=3765211 2023-12-03T11:30:46.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.885+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=B4TDEM17M3LX type=del size=1249731 2023-12-03T11:30:46.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.886+0000 7fc553ded780 1 < canceled operation key=TQRZT37AB9WC tag=Q17M5XODZTEU type=del size=1704092 2023-12-03T11:30:46.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.889+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~HGZCA1AIPBBH.0 size=798521 2023-12-03T11:30:46.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.889+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=HGZCA1AIPBBH tag=KXGDUNX38MBH type=write size=1847097 2023-12-03T11:30:46.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.890+0000 7fc553ded780 1 < canceled operation key=588S2ASN9I0W tag=8E5JIDQJFCUI type=del size=142688 2023-12-03T11:30:46.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.892+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~TSHHK4J5Z3JY.0 size=967869 2023-12-03T11:30:46.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.892+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=TSHHK4J5Z3JY tag=9JXJ07JUVXB6 type=write size=2016445 2023-12-03T11:30:46.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.893+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=9O4GD8UQXAY2 type=del 2023-12-03T11:30:46.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.895+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.0 size=1048576 2023-12-03T11:30:46.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.896+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.1 size=1048576 2023-12-03T11:30:46.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.897+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.2 size=97238 2023-12-03T11:30:46.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.897+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=V8VRH6UFMNI1 tag=Q2XF297JVDNQ type=write size=3242966 2023-12-03T11:30:46.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.899+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=VNART804203O type=del 2023-12-03T11:30:46.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.900+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=AVLCUXU1F3UR type=del size=1390992 2023-12-03T11:30:46.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.901+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=VY3UDOEK5TWN type=del 2023-12-03T11:30:46.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.903+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~03Q2GPU60QJZ.0 size=149876 2023-12-03T11:30:46.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.903+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=03Q2GPU60QJZ tag=SO6S1LFBC395 type=write size=1198452 2023-12-03T11:30:46.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.906+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.0 size=1048576 2023-12-03T11:30:46.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.907+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.1 size=1048576 2023-12-03T11:30:46.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.908+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~AVNDZH6MHY8L.2 size=154028 2023-12-03T11:30:46.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.910+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=AVNDZH6MHY8L tag=BZXSY65GAL21 type=write size=3299756 2023-12-03T11:30:46.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.912+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=DV6DEI88NLL9 type=del size=1961541 2023-12-03T11:30:46.914 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.913+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=FW74550PX5RR type=del 2023-12-03T11:30:46.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.914+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=2HC1AHI59OKT type=del size=3860125 2023-12-03T11:30:46.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.916+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=Z9TP839G1BAY type=del 2023-12-03T11:30:46.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.918+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.0 size=1048576 2023-12-03T11:30:46.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.919+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.1 size=1048576 2023-12-03T11:30:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.920+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.2 size=53832 2023-12-03T11:30:46.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.920+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=79YQFQ2I2AF5 tag=K0XAU8F86208 type=write size=3199560 2023-12-03T11:30:46.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.921+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~HF26HVKBYK4W.0 size=1048576 2023-12-03T11:30:46.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.921+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~HF26HVKBYK4W.1 size=242855 2023-12-03T11:30:46.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.923+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=HF26HVKBYK4W tag=8W7V5M0LGIPX type=write size=2340007 2023-12-03T11:30:46.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.924+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=SPQE5VIJM5FX type=write size=847007 2023-12-03T11:30:46.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.925+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~ZJ26WIU7X3ZU.0 size=707391 2023-12-03T11:30:46.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.927+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=ZJ26WIU7X3ZU tag=4JVFXD9O76LJ type=write size=1755967 2023-12-03T11:30:46.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.930+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.0 size=1048576 2023-12-03T11:30:46.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.931+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.1 size=1048576 2023-12-03T11:30:46.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.932+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.2 size=340682 2023-12-03T11:30:46.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.932+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=KHJIG0Q2J92G tag=UZ2L2SLBYPDA type=write size=3486410 2023-12-03T11:30:46.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.933+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.0 size=1048576 2023-12-03T11:30:46.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.934+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.1 size=1048576 2023-12-03T11:30:46.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.935+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~MK6LCSBC640M.2 size=673325 2023-12-03T11:30:46.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.937+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=MK6LCSBC640M tag=8YGN2KWAXIV2 type=write size=3819053 2023-12-03T11:30:46.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.938+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=L3FVQ7H9TSI5 type=del size=1991333 2023-12-03T11:30:46.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.939+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~XE4OL2HO81IY.0 size=642152 2023-12-03T11:30:46.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.941+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=XE4OL2HO81IY tag=J4CK63FKL2HD type=write size=1690728 2023-12-03T11:30:46.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.943+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=1OGXNB013FW5 type=del size=1643219 2023-12-03T11:30:46.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.944+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=7W3SGR5XN8GV type=del 2023-12-03T11:30:46.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.945+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=LQDTXMIWUQPR type=del size=2182806 2023-12-03T11:30:46.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.947+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~VXM97O9N55XB.0 size=1048576 2023-12-03T11:30:46.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.948+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~VXM97O9N55XB.1 size=583505 2023-12-03T11:30:46.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.950+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=VXM97O9N55XB tag=QFKVD0NF1JUJ type=write size=2680657 2023-12-03T11:30:46.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.953+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~QKKG4DIS7H1E.0 size=1048576 2023-12-03T11:30:46.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.954+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~QKKG4DIS7H1E.1 size=804746 2023-12-03T11:30:46.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.954+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=QKKG4DIS7H1E tag=YA3WAP0J09EM type=write size=2901898 2023-12-03T11:30:46.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.955+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=NVQLYZUYFFX6 type=del 2023-12-03T11:30:46.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.956+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=O9QHTHW6P2ZD type=del size=2697994 2023-12-03T11:30:46.958 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.957+0000 7fc553ded780 1 < canceled operation key=TQRZT37AB9WC tag=B4TDEM17M3LX type=del size=1249731 2023-12-03T11:30:46.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.959+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=BZ2ADRJ3CO9P type=del size=4157842 2023-12-03T11:30:46.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.960+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~HGZCA1AIPBBH.0 size=798521 2023-12-03T11:30:46.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.962+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=HGZCA1AIPBBH tag=KXGDUNX38MBH type=write size=1847097 2023-12-03T11:30:46.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.963+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=2WP2805QCRMJ type=del size=1687320 2023-12-03T11:30:46.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.964+0000 7fc553ded780 1 < canceled part key=_multipart_ZUNOFY5N3J3H.2~TSHHK4J5Z3JY.0 size=967869 2023-12-03T11:30:46.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.965+0000 7fc553ded780 1 < canceled multipart upload key=ZUNOFY5N3J3H upload=TSHHK4J5Z3JY tag=9JXJ07JUVXB6 type=write size=2016445 2023-12-03T11:30:46.968 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.967+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=M1MRMK2ZEPO1 type=write size=934200 2023-12-03T11:30:46.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.968+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.0 size=1048576 2023-12-03T11:30:46.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.969+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.1 size=1048576 2023-12-03T11:30:46.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.969+0000 7fc553ded780 1 < canceled part key=_multipart_WLTMDGRBF4C9.2~V8VRH6UFMNI1.2 size=97238 2023-12-03T11:30:46.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.971+0000 7fc553ded780 1 < canceled multipart upload key=WLTMDGRBF4C9 upload=V8VRH6UFMNI1 tag=Q2XF297JVDNQ type=write size=3242966 2023-12-03T11:30:46.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.974+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~0ILXC6DCY7UJ.0 size=970806 2023-12-03T11:30:46.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.974+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=0ILXC6DCY7UJ tag=LTMCTYEHFKPZ type=write size=2019382 2023-12-03T11:30:46.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.975+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=AVLCUXU1F3UR type=del size=1390992 2023-12-03T11:30:46.979 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.978+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=MOMWZULO3VZY type=del size=2636208 2023-12-03T11:30:46.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.979+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~03Q2GPU60QJZ.0 size=149876 2023-12-03T11:30:46.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.982+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=03Q2GPU60QJZ tag=SO6S1LFBC395 type=write size=1198452 2023-12-03T11:30:46.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.983+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=UURB28ERJMWO type=del size=2369221 2023-12-03T11:30:46.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.986+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=DV6DEI88NLL9 type=del size=1961541 2023-12-03T11:30:46.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.987+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=650QARL39R6A type=del size=3049520 2023-12-03T11:30:46.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.989+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=2HC1AHI59OKT type=del 2023-12-03T11:30:46.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.991+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~C21EE7HQR09O.0 size=1048576 2023-12-03T11:30:46.993 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.992+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~C21EE7HQR09O.1 size=737190 2023-12-03T11:30:46.993 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.992+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=C21EE7HQR09O tag=UORGX23ENJVP type=write size=2834342 2023-12-03T11:30:46.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.993+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.0 size=1048576 2023-12-03T11:30:46.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.995+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.1 size=1048576 2023-12-03T11:30:46.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.996+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~79YQFQ2I2AF5.2 size=53832 2023-12-03T11:30:46.999 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.998+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=79YQFQ2I2AF5 tag=K0XAU8F86208 type=write size=3199560 2023-12-03T11:30:47.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:46.999+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=E6NVNIS1PTEJ type=write size=1018379 2023-12-03T11:30:47.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.000+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=SPQE5VIJM5FX type=write size=847007 2023-12-03T11:30:47.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.002+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~X62VNCP77XFO.0 size=1048576 2023-12-03T11:30:47.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.003+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~X62VNCP77XFO.1 size=15838 2023-12-03T11:30:47.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.003+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=X62VNCP77XFO tag=QEK2VKMDE9Q4 type=write size=2112990 2023-12-03T11:30:47.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.005+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.0 size=1048576 2023-12-03T11:30:47.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.005+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.1 size=1048576 2023-12-03T11:30:47.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.006+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~KHJIG0Q2J92G.2 size=340682 2023-12-03T11:30:47.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.007+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=KHJIG0Q2J92G tag=UZ2L2SLBYPDA type=write size=3486410 2023-12-03T11:30:47.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.009+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=D57Z6XJPADVX type=del size=4043111 2023-12-03T11:30:47.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.010+0000 7fc553ded780 1 < completed delete operation key=0H5OZMBMLQFJ tag=L3FVQ7H9TSI5 type=del 2023-12-03T11:30:47.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.012+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~VWO9OY303K3G.0 size=1048576 2023-12-03T11:30:47.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.013+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~VWO9OY303K3G.1 size=253178 2023-12-03T11:30:47.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.013+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=VWO9OY303K3G tag=UR7P2M04YXSN type=write size=2350330 2023-12-03T11:30:47.015 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.014+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=1OGXNB013FW5 type=del 2023-12-03T11:30:47.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.015+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=H1M4V2X0KO16 type=del size=2911471 2023-12-03T11:30:47.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.016+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=LQDTXMIWUQPR type=del 2023-12-03T11:30:47.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.018+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=7A5RSGS4MKSI type=del size=19606 2023-12-03T11:30:47.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.019+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~QKKG4DIS7H1E.0 size=1048576 2023-12-03T11:30:47.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.020+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~QKKG4DIS7H1E.1 size=804746 2023-12-03T11:30:47.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.022+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=QKKG4DIS7H1E tag=YA3WAP0J09EM type=write size=2901898 2023-12-03T11:30:47.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.023+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=XD9QILPUNCS3 type=del size=3044960 2023-12-03T11:30:47.025 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.024+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=O9QHTHW6P2ZD type=del 2023-12-03T11:30:47.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.025+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=6S34YV8IJ6UE type=write size=801510 2023-12-03T11:30:47.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.026+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=BZ2ADRJ3CO9P type=del 2023-12-03T11:30:47.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.028+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=IS510M4SWQXY type=del size=1052509 2023-12-03T11:30:47.031 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.030+0000 7fc553ded780 1 < suggested remove operation key=I3U08QX8MLBD tag=2WP2805QCRMJ type=del 2023-12-03T11:30:47.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.031+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=BMWHRTSVCHY6 type=del size=2467939 2023-12-03T11:30:47.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.032+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=M1MRMK2ZEPO1 type=write size=934200 2023-12-03T11:30:47.034 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.033+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=XILUNAA23NEA type=write size=496656 2023-12-03T11:30:47.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.035+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~0ILXC6DCY7UJ.0 size=970806 2023-12-03T11:30:47.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.037+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=0ILXC6DCY7UJ tag=LTMCTYEHFKPZ type=write size=2019382 2023-12-03T11:30:47.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.038+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=ANTK8WUFNDEV type=del size=1259567 2023-12-03T11:30:47.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.039+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=MOMWZULO3VZY type=del 2023-12-03T11:30:47.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.041+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~NLMV2D2GXP9K.0 size=1048576 2023-12-03T11:30:47.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.042+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~NLMV2D2GXP9K.1 size=6785 2023-12-03T11:30:47.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.042+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=NLMV2D2GXP9K tag=LLIA85YNDKPZ type=write size=2103937 2023-12-03T11:30:47.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.044+0000 7fc553ded780 1 < suggested remove operation key=X741JIRT7RIG tag=UURB28ERJMWO type=del 2023-12-03T11:30:47.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.046+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=B95R69UUMHKQ type=del size=2338496 2023-12-03T11:30:47.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.048+0000 7fc553ded780 1 < suggested remove operation key=FC4WU6SMNEWV tag=650QARL39R6A type=del 2023-12-03T11:30:47.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.050+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~MQ6YKL5532WI.0 size=1048576 2023-12-03T11:30:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.051+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~MQ6YKL5532WI.1 size=231881 2023-12-03T11:30:47.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.051+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=MQ6YKL5532WI tag=RSWQ2VR8J0IL type=write size=2329033 2023-12-03T11:30:47.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.053+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~C21EE7HQR09O.0 size=1048576 2023-12-03T11:30:47.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.054+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~C21EE7HQR09O.1 size=737190 2023-12-03T11:30:47.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.056+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=C21EE7HQR09O tag=UORGX23ENJVP type=write size=2834342 2023-12-03T11:30:47.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.057+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=STBWFFJW3FY6 type=del size=3184977 2023-12-03T11:30:47.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.058+0000 7fc553ded780 1 < completed write operation key=EK7V6U9DSLD0 tag=E6NVNIS1PTEJ type=write size=1018379 2023-12-03T11:30:47.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.061+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~H64GOVINBBWQ.0 size=1048576 2023-12-03T11:30:47.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.062+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~H64GOVINBBWQ.1 size=164640 2023-12-03T11:30:47.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.062+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=H64GOVINBBWQ tag=GUJKWIEK1HF2 type=write size=2261792 2023-12-03T11:30:47.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.063+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~X62VNCP77XFO.0 size=1048576 2023-12-03T11:30:47.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.063+0000 7fc553ded780 1 < canceled part key=_multipart_UKAL8FCVIRSH.2~X62VNCP77XFO.1 size=15838 2023-12-03T11:30:47.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.065+0000 7fc553ded780 1 < canceled multipart upload key=UKAL8FCVIRSH upload=X62VNCP77XFO tag=QEK2VKMDE9Q4 type=write size=2112990 2023-12-03T11:30:47.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.066+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=YV2GKCEUHG3O type=del size=2125911 2023-12-03T11:30:47.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.069+0000 7fc553ded780 1 < suggested update operation key=PL22GPUFN3O0 tag=D57Z6XJPADVX type=del size=4043111 2023-12-03T11:30:47.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.070+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=99NA8RIOQF4S type=del size=1955495 2023-12-03T11:30:47.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.071+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~VWO9OY303K3G.0 size=1048576 2023-12-03T11:30:47.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.072+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~VWO9OY303K3G.1 size=253178 2023-12-03T11:30:47.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.074+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=VWO9OY303K3G tag=UR7P2M04YXSN type=write size=2350330 2023-12-03T11:30:47.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.076+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=BLNV7PFYP3UY type=write size=976656 2023-12-03T11:30:47.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.077+0000 7fc553ded780 1 < completed delete operation key=588S2ASN9I0W tag=H1M4V2X0KO16 type=del 2023-12-03T11:30:47.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.078+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=O2SSMVX4WDZ0 type=del size=1851973 2023-12-03T11:30:47.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.079+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=7A5RSGS4MKSI type=del 2023-12-03T11:30:47.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.080+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=ECLFKQJAWM15 type=del size=3308670 2023-12-03T11:30:47.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.083+0000 7fc553ded780 1 < suggested remove operation key=WOKG1DISIKK0 tag=XD9QILPUNCS3 type=del 2023-12-03T11:30:47.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.084+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=FUA91OB8GXZ9 type=del size=1513439 2023-12-03T11:30:47.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.085+0000 7fc553ded780 1 < completed write operation key=FP2UZMDKYTD1 tag=6S34YV8IJ6UE type=write size=801510 2023-12-03T11:30:47.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.086+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=2F21DEK61KO1 type=del size=1912119 2023-12-03T11:30:47.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.088+0000 7fc553ded780 1 < canceled operation key=0H5OZMBMLQFJ tag=IS510M4SWQXY type=del size=1052509 2023-12-03T11:30:47.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.090+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~M6V6CO1H0OD4.0 size=1048576 2023-12-03T11:30:47.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.091+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~M6V6CO1H0OD4.1 size=732282 2023-12-03T11:30:47.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.091+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=M6V6CO1H0OD4 tag=87FOP1QX959I type=write size=2829434 2023-12-03T11:30:47.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.092+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=BMWHRTSVCHY6 type=del 2023-12-03T11:30:47.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.094+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=WXHQ5MWUBVOF type=del size=875482 2023-12-03T11:30:47.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.095+0000 7fc553ded780 -1 < no bi entry to suggest for operation key=FC4WU6SMNEWV tag=XILUNAA23NEA type=write size=496656: (2) No such file or directory 2023-12-03T11:30:47.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.098+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~FOWAWSXWLXVJ.0 size=483398 2023-12-03T11:30:47.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.098+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=FOWAWSXWLXVJ tag=IVPZ98C3MARQ type=write size=1531974 2023-12-03T11:30:47.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.100+0000 7fc553ded780 -1 < failed to complete operation key=PL22GPUFN3O0 tag=ANTK8WUFNDEV type=del size=1259567: (22) Invalid argument 2023-12-03T11:30:47.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.102+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~FMRT736FLRE0.0 size=1048576 2023-12-03T11:30:47.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.103+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~FMRT736FLRE0.1 size=868354 2023-12-03T11:30:47.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.103+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=FMRT736FLRE0 tag=1C645EEXB7M5 type=write size=2965506 2023-12-03T11:30:47.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.104+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~NLMV2D2GXP9K.0 size=1048576 2023-12-03T11:30:47.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.105+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~NLMV2D2GXP9K.1 size=6785 2023-12-03T11:30:47.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.107+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=NLMV2D2GXP9K tag=LLIA85YNDKPZ type=write size=2103937 2023-12-03T11:30:47.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.108+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=24EFF9DOANZY type=del size=1279442 2023-12-03T11:30:47.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.109+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=B95R69UUMHKQ type=del 2023-12-03T11:30:47.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.110+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=1MX3WLQ3J0RP type=del size=3571141 2023-12-03T11:30:47.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.112+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~MQ6YKL5532WI.0 size=1048576 2023-12-03T11:30:47.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.112+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~MQ6YKL5532WI.1 size=231881 2023-12-03T11:30:47.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.114+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=MQ6YKL5532WI tag=RSWQ2VR8J0IL type=write size=2329033 2023-12-03T11:30:47.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.116+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~WRYS3G4J1B3R.0 size=147111 2023-12-03T11:30:47.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.116+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=WRYS3G4J1B3R tag=T4EXQOCJNFMF type=write size=1195687 2023-12-03T11:30:47.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.118+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=STBWFFJW3FY6 type=del 2023-12-03T11:30:47.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.119+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=WHYPF75B71LL type=del size=2668385 2023-12-03T11:30:47.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.120+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~H64GOVINBBWQ.0 size=1048576 2023-12-03T11:30:47.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.121+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~H64GOVINBBWQ.1 size=164640 2023-12-03T11:30:47.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.123+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=H64GOVINBBWQ tag=GUJKWIEK1HF2 type=write size=2261792 2023-12-03T11:30:47.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.125+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=HSARVGST1HXG type=del size=27339 2023-12-03T11:30:47.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.126+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=YV2GKCEUHG3O type=del 2023-12-03T11:30:47.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.128+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~1G8UYU1AA27E.0 size=1048576 2023-12-03T11:30:47.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.129+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~1G8UYU1AA27E.1 size=39280 2023-12-03T11:30:47.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.129+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=1G8UYU1AA27E tag=RLQDNVYEY6GL type=write size=2136432 2023-12-03T11:30:47.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.131+0000 7fc553ded780 1 < suggested update operation key=RO0KGI2QMA71 tag=99NA8RIOQF4S type=del size=1955495 2023-12-03T11:30:47.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.134+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~DS3EVM1C1UUT.0 size=544982 2023-12-03T11:30:47.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.134+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=DS3EVM1C1UUT tag=FS6VX465M10I type=write size=1593558 2023-12-03T11:30:47.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.135+0000 7fc553ded780 1 < canceled operation key=G1OBBD75M5ZQ tag=BLNV7PFYP3UY type=write size=976656 2023-12-03T11:30:47.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.137+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~I2RSMQP5V2QR.0 size=1048576 2023-12-03T11:30:47.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.138+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~I2RSMQP5V2QR.1 size=592977 2023-12-03T11:30:47.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.138+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=I2RSMQP5V2QR tag=N895NJO1ESEK type=write size=2690129 2023-12-03T11:30:47.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.139+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=O2SSMVX4WDZ0 type=del 2023-12-03T11:30:47.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.140+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=T7BMK9TOSSW4 type=del size=2446247 2023-12-03T11:30:47.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.143+0000 7fc553ded780 1 < suggested remove operation key=8GRA2MUBGWJI tag=ECLFKQJAWM15 type=del 2023-12-03T11:30:47.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.145+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.0 size=1048576 2023-12-03T11:30:47.147 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.146+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.1 size=1048576 2023-12-03T11:30:47.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.147+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.2 size=177473 2023-12-03T11:30:47.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.147+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=O2U9ZGD60WGE tag=CDEN0TVTZLO3 type=write size=3323201 2023-12-03T11:30:47.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.148+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=FUA91OB8GXZ9 type=del 2023-12-03T11:30:47.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.149+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=II1NHGLMGQBV type=write size=460603 2023-12-03T11:30:47.151 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.150+0000 7fc553ded780 1 < canceled operation key=ICDF1EYNI5IU tag=2F21DEK61KO1 type=del size=1912119 2023-12-03T11:30:47.153 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.152+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~S64THVIF969B.0 size=160231 2023-12-03T11:30:47.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.152+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=S64THVIF969B tag=9J5Y6R6CZZ00 type=write size=1208807 2023-12-03T11:30:47.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.154+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~M6V6CO1H0OD4.0 size=1048576 2023-12-03T11:30:47.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.155+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~M6V6CO1H0OD4.1 size=732282 2023-12-03T11:30:47.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.157+0000 7fc553ded780 1 < completed multipart upload key=YE73HBF64CS7 upload=M6V6CO1H0OD4 tag=87FOP1QX959I type=write size=2829434 2023-12-03T11:30:47.159 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.158+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=X45DO1UBB6H4 type=del size=1141724 2023-12-03T11:30:47.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.159+0000 7fc553ded780 1 < canceled operation key=LRN5NYUZ5JT9 tag=WXHQ5MWUBVOF type=del size=875482 2023-12-03T11:30:47.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.161+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=YGJYRZDY7ULO type=del size=1104567 2023-12-03T11:30:47.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.162+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~FOWAWSXWLXVJ.0 size=483398 2023-12-03T11:30:47.165 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.164+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=FOWAWSXWLXVJ tag=IVPZ98C3MARQ type=write size=1531974 2023-12-03T11:30:47.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.165+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=YUI6LHTETH5H type=del size=1024078 2023-12-03T11:30:47.167 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.166+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~FMRT736FLRE0.0 size=1048576 2023-12-03T11:30:47.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.167+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~FMRT736FLRE0.1 size=868354 2023-12-03T11:30:47.170 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.170+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=FMRT736FLRE0 tag=1C645EEXB7M5 type=write size=2965506 2023-12-03T11:30:47.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.171+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=FEIRBRGPEJ93 type=del size=2076248 2023-12-03T11:30:47.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.173+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=24EFF9DOANZY type=del 2023-12-03T11:30:47.176 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.175+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~WACQZ7ITWI6V.0 size=1048576 2023-12-03T11:30:47.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.176+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~WACQZ7ITWI6V.1 size=387794 2023-12-03T11:30:47.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.176+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=WACQZ7ITWI6V tag=0QMD3LBR2A9C type=write size=2484946 2023-12-03T11:30:47.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.177+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=1MX3WLQ3J0RP type=del 2023-12-03T11:30:47.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.179+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=PTV7O47XR90Y type=del size=1930682 2023-12-03T11:30:47.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.180+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~WRYS3G4J1B3R.0 size=147111 2023-12-03T11:30:47.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.182+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=WRYS3G4J1B3R tag=T4EXQOCJNFMF type=write size=1195687 2023-12-03T11:30:47.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.183+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=1DRIL8L5TRW5 type=del size=3078444 2023-12-03T11:30:47.185 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.184+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=WHYPF75B71LL type=del 2023-12-03T11:30:47.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.187+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.0 size=1048576 2023-12-03T11:30:47.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.188+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.1 size=1048576 2023-12-03T11:30:47.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.189+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.2 size=306565 2023-12-03T11:30:47.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.189+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=XN1NDM7PVAP3 tag=955S0CXMRBPB type=write size=3452293 2023-12-03T11:30:47.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.190+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=HSARVGST1HXG type=del 2023-12-03T11:30:47.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.192+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~RE1T84TY0MBW.0 size=268089 2023-12-03T11:30:47.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.192+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=RE1T84TY0MBW tag=ZRJI1DB8W2LX type=write size=1316665 2023-12-03T11:30:47.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.193+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~1G8UYU1AA27E.0 size=1048576 2023-12-03T11:30:47.195 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.194+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~1G8UYU1AA27E.1 size=39280 2023-12-03T11:30:47.198 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.197+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=1G8UYU1AA27E tag=RLQDNVYEY6GL type=write size=2136432 2023-12-03T11:30:47.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.198+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=9N2CSRSOJ1LU type=write size=468153 2023-12-03T11:30:47.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.199+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~DS3EVM1C1UUT.0 size=544982 2023-12-03T11:30:47.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.201+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=DS3EVM1C1UUT tag=FS6VX465M10I type=write size=1593558 2023-12-03T11:30:47.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.203+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.0 size=1048576 2023-12-03T11:30:47.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.204+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.1 size=1048576 2023-12-03T11:30:47.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.206+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.2 size=934592 2023-12-03T11:30:47.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.206+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=2VRLVRBP3A6O tag=KLFSJFYM178G type=write size=4080320 2023-12-03T11:30:47.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.207+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~I2RSMQP5V2QR.0 size=1048576 2023-12-03T11:30:47.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.208+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~I2RSMQP5V2QR.1 size=592977 2023-12-03T11:30:47.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.211+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=I2RSMQP5V2QR tag=N895NJO1ESEK type=write size=2690129 2023-12-03T11:30:47.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.216+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~TJOCF4TH0V19.0 size=789351 2023-12-03T11:30:47.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.216+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=TJOCF4TH0V19 tag=V9YXR9ZWN3RM type=write size=1837927 2023-12-03T11:30:47.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.217+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=T7BMK9TOSSW4 type=del 2023-12-03T11:30:47.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.218+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=PEEPDNXOR41B type=del size=3378102 2023-12-03T11:30:47.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.219+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.0 size=1048576 2023-12-03T11:30:47.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.219+0000 7fc553ded780 1 < canceled part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.1 size=1048576 2023-12-03T11:30:47.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.220+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~O2U9ZGD60WGE.2 size=177473 2023-12-03T11:30:47.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.222+0000 7fc553ded780 1 < canceled multipart upload key=WLTMDGRBF4C9 upload=O2U9ZGD60WGE tag=CDEN0TVTZLO3 type=write size=3323201 2023-12-03T11:30:47.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.223+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=UVST41QRX6F9 type=del size=1136586 2023-12-03T11:30:47.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.224+0000 7fc553ded780 1 < completed write operation key=WDV4CW863V33 tag=II1NHGLMGQBV type=write size=460603 2023-12-03T11:30:47.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.226+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=FCAYEKWTBZSK type=del size=2383779 2023-12-03T11:30:47.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.227+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~S64THVIF969B.0 size=160231 2023-12-03T11:30:47.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.229+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=S64THVIF969B tag=9J5Y6R6CZZ00 type=write size=1208807 2023-12-03T11:30:47.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.230+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=BRV9VX416I7R type=del size=1122508 2023-12-03T11:30:47.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.232+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=X45DO1UBB6H4 type=del 2023-12-03T11:30:47.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.234+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.0 size=1048576 2023-12-03T11:30:47.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.235+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.1 size=1048576 2023-12-03T11:30:47.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.236+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.2 size=74527 2023-12-03T11:30:47.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.236+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=50YT5AKIFARX tag=JPSUHZ8ZZ70D type=write size=3220255 2023-12-03T11:30:47.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.237+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=YGJYRZDY7ULO type=del 2023-12-03T11:30:47.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.240+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~6XMADBP9O4Z1.0 size=719694 2023-12-03T11:30:47.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.240+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=6XMADBP9O4Z1 tag=30VFFOFDD1GD type=write size=1768270 2023-12-03T11:30:47.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.241+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=YUI6LHTETH5H type=del 2023-12-03T11:30:47.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.243+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~00R2XIQZ65Z6.0 size=82378 2023-12-03T11:30:47.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.243+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=00R2XIQZ65Z6 tag=2WE7UR595G65 type=write size=1130954 2023-12-03T11:30:47.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.244+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=FEIRBRGPEJ93 type=del 2023-12-03T11:30:47.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.245+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=PCOSSCS38YIC type=write size=217315 2023-12-03T11:30:47.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.246+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~WACQZ7ITWI6V.0 size=1048576 2023-12-03T11:30:47.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.246+0000 7fc553ded780 1 < canceled part key=_multipart_WDV4CW863V33.2~WACQZ7ITWI6V.1 size=387794 2023-12-03T11:30:47.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.248+0000 7fc553ded780 1 < canceled multipart upload key=WDV4CW863V33 upload=WACQZ7ITWI6V tag=0QMD3LBR2A9C type=write size=2484946 2023-12-03T11:30:47.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.249+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=1XV86FJ12J29 type=del size=1639129 2023-12-03T11:30:47.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.251+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=PTV7O47XR90Y type=del 2023-12-03T11:30:47.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.253+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~KJ22E6YOY3II.0 size=688547 2023-12-03T11:30:47.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.253+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=KJ22E6YOY3II tag=4551DPE87EJS type=write size=1737123 2023-12-03T11:30:47.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.255+0000 7fc553ded780 1 < suggested remove operation key=6OW429S51UWN tag=1DRIL8L5TRW5 type=del 2023-12-03T11:30:47.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.256+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=JFJW2JWDIMJ2 type=del size=2188893 2023-12-03T11:30:47.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.258+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.0 size=1048576 2023-12-03T11:30:47.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.259+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.1 size=1048576 2023-12-03T11:30:47.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.259+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~XN1NDM7PVAP3.2 size=306565 2023-12-03T11:30:47.262 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.261+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=XN1NDM7PVAP3 tag=955S0CXMRBPB type=write size=3452293 2023-12-03T11:30:47.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.262+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=ZRO0W9DSJ1F7 type=write size=760109 2023-12-03T11:30:47.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.263+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~RE1T84TY0MBW.0 size=268089 2023-12-03T11:30:47.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.265+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=RE1T84TY0MBW tag=ZRJI1DB8W2LX type=write size=1316665 2023-12-03T11:30:47.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.266+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=90F44K4B3DYM type=write size=945038 2023-12-03T11:30:47.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.268+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=9N2CSRSOJ1LU type=write size=468153 2023-12-03T11:30:47.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.269+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=U3OO0YKZXUDP type=del size=3873574 2023-12-03T11:30:47.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.270+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.0 size=1048576 2023-12-03T11:30:47.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.271+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.1 size=1048576 2023-12-03T11:30:47.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.272+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~2VRLVRBP3A6O.2 size=934592 2023-12-03T11:30:47.275 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.274+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=2VRLVRBP3A6O tag=KLFSJFYM178G type=write size=4080320 2023-12-03T11:30:47.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.276+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~SMBD3A0KANQY.0 size=1048576 2023-12-03T11:30:47.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.277+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~SMBD3A0KANQY.1 size=591663 2023-12-03T11:30:47.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.277+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=SMBD3A0KANQY tag=N9VL8UOF0ZBR type=write size=2688815 2023-12-03T11:30:47.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.279+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~TJOCF4TH0V19.0 size=789351 2023-12-03T11:30:47.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.281+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=TJOCF4TH0V19 tag=V9YXR9ZWN3RM type=write size=1837927 2023-12-03T11:30:47.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.283+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~ADOQGZQ6CN9T.0 size=1048576 2023-12-03T11:30:47.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.284+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~ADOQGZQ6CN9T.1 size=547146 2023-12-03T11:30:47.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.284+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=ADOQGZQ6CN9T tag=EWM4EBSSTUAU type=write size=2644298 2023-12-03T11:30:47.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.286+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=PEEPDNXOR41B type=del 2023-12-03T11:30:47.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.288+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~34ALIWDPTXE1.0 size=1048576 2023-12-03T11:30:47.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.289+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~34ALIWDPTXE1.1 size=707257 2023-12-03T11:30:47.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.289+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=34ALIWDPTXE1 tag=FQYO3P17MN5R type=write size=2804409 2023-12-03T11:30:47.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.291+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=UVST41QRX6F9 type=del 2023-12-03T11:30:47.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.293+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.0 size=1048576 2023-12-03T11:30:47.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.294+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.1 size=1048576 2023-12-03T11:30:47.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.295+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.2 size=447559 2023-12-03T11:30:47.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.295+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=PP17ZI3MU8PA tag=UIRGTDA7IQWT type=write size=3593287 2023-12-03T11:30:47.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.296+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=FCAYEKWTBZSK type=del 2023-12-03T11:30:47.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.298+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~2LJHLL8ZMNVT.0 size=605565 2023-12-03T11:30:47.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.298+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=2LJHLL8ZMNVT tag=WPKHM5FALUOF type=write size=1654141 2023-12-03T11:30:47.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.299+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=BRV9VX416I7R type=del 2023-12-03T11:30:47.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.301+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=K4448DOXNRJG type=write size=648181 2023-12-03T11:30:47.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.302+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.0 size=1048576 2023-12-03T11:30:47.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.303+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.1 size=1048576 2023-12-03T11:30:47.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.304+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~50YT5AKIFARX.2 size=74527 2023-12-03T11:30:47.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.306+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=50YT5AKIFARX tag=JPSUHZ8ZZ70D type=write size=3220255 2023-12-03T11:30:47.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.307+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=97OCPJ2QRJG6 type=write size=1015715 2023-12-03T11:30:47.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.309+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~6XMADBP9O4Z1.0 size=719694 2023-12-03T11:30:47.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.311+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=6XMADBP9O4Z1 tag=30VFFOFDD1GD type=write size=1768270 2023-12-03T11:30:47.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.312+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=DQSJZHHPGIZ8 type=del size=1499106 2023-12-03T11:30:47.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.313+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~00R2XIQZ65Z6.0 size=82378 2023-12-03T11:30:47.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.315+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=00R2XIQZ65Z6 tag=2WE7UR595G65 type=write size=1130954 2023-12-03T11:30:47.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.316+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=2ZVN9ICMSGQB type=del size=3797078 2023-12-03T11:30:47.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.317+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=PCOSSCS38YIC type=write size=217315 2023-12-03T11:30:47.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.319+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=8KP0XIGKWVPO type=del size=3292364 2023-12-03T11:30:47.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.320+0000 7fc553ded780 1 < canceled operation key=WLTMDGRBF4C9 tag=1XV86FJ12J29 type=del size=1639129 2023-12-03T11:30:47.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.323+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~98WRP2N0A4C0.0 size=1048576 2023-12-03T11:30:47.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.325+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~98WRP2N0A4C0.1 size=209633 2023-12-03T11:30:47.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.325+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=98WRP2N0A4C0 tag=BRYA49YDM9MV type=write size=2306785 2023-12-03T11:30:47.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.326+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~KJ22E6YOY3II.0 size=688547 2023-12-03T11:30:47.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.328+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=KJ22E6YOY3II tag=4551DPE87EJS type=write size=1737123 2023-12-03T11:30:47.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.329+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=B5V5RY7DSS1Z type=del size=521257 2023-12-03T11:30:47.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.331+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=JFJW2JWDIMJ2 type=del 2023-12-03T11:30:47.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.332+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=1J0YWNDPOGHQ type=del size=2002390 2023-12-03T11:30:47.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.334+0000 7fc553ded780 1 < suggested update operation key=X741JIRT7RIG tag=ZRO0W9DSJ1F7 type=write size=760109 2023-12-03T11:30:47.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.336+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~0JMSS9ANLOMU.0 size=1048576 2023-12-03T11:30:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.337+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~0JMSS9ANLOMU.1 size=425324 2023-12-03T11:30:47.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.337+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=0JMSS9ANLOMU tag=PBCU0Q2JJN4V type=write size=2522476 2023-12-03T11:30:47.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.339+0000 7fc553ded780 1 < canceled operation key=C2TPF9Q5Z7XQ tag=90F44K4B3DYM type=write size=945038 2023-12-03T11:30:47.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.341+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.0 size=1048576 2023-12-03T11:30:47.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.342+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.1 size=1048576 2023-12-03T11:30:47.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.343+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.2 size=476729 2023-12-03T11:30:47.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.343+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=1E93J39WJNHS tag=TISQYRMDQJJT type=write size=3622457 2023-12-03T11:30:47.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.345+0000 7fc553ded780 1 < suggested remove operation key=4TWM1QM1OHOF tag=U3OO0YKZXUDP type=del 2023-12-03T11:30:47.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.347+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4XPXU61630XC.0 size=1048576 2023-12-03T11:30:47.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.348+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~4XPXU61630XC.1 size=260089 2023-12-03T11:30:47.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.348+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=4XPXU61630XC tag=QC32ABT1EUVS type=write size=2357241 2023-12-03T11:30:47.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.350+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~SMBD3A0KANQY.0 size=1048576 2023-12-03T11:30:47.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.351+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~SMBD3A0KANQY.1 size=591663 2023-12-03T11:30:47.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.353+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=SMBD3A0KANQY tag=N9VL8UOF0ZBR type=write size=2688815 2023-12-03T11:30:47.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.354+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=Z8CU3M68966C type=write size=737870 2023-12-03T11:30:47.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.355+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~ADOQGZQ6CN9T.0 size=1048576 2023-12-03T11:30:47.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.357+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~ADOQGZQ6CN9T.1 size=547146 2023-12-03T11:30:47.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.359+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=ADOQGZQ6CN9T tag=EWM4EBSSTUAU type=write size=2644298 2023-12-03T11:30:47.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.360+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=0PYDQCVJELX3 type=del size=520296 2023-12-03T11:30:47.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.361+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~34ALIWDPTXE1.0 size=1048576 2023-12-03T11:30:47.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.362+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~34ALIWDPTXE1.1 size=707257 2023-12-03T11:30:47.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.364+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=34ALIWDPTXE1 tag=FQYO3P17MN5R type=write size=2804409 2023-12-03T11:30:47.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.365+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=05QFD7EWRB2Q type=del size=605352 2023-12-03T11:30:47.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.366+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.0 size=1048576 2023-12-03T11:30:47.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.367+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.1 size=1048576 2023-12-03T11:30:47.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.368+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~PP17ZI3MU8PA.2 size=447559 2023-12-03T11:30:47.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.371+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=PP17ZI3MU8PA tag=UIRGTDA7IQWT type=write size=3593287 2023-12-03T11:30:47.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.373+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.0 size=1048576 2023-12-03T11:30:47.375 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.374+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.1 size=1048576 2023-12-03T11:30:47.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.375+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.2 size=445029 2023-12-03T11:30:47.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.375+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=M95SAWKC1AE1 tag=WE0QDJDFMG0D type=write size=3590757 2023-12-03T11:30:47.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.376+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~2LJHLL8ZMNVT.0 size=605565 2023-12-03T11:30:47.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.378+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=2LJHLL8ZMNVT tag=WPKHM5FALUOF type=write size=1654141 2023-12-03T11:30:47.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.379+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=4G1O9VED41Q1 type=del size=2431662 2023-12-03T11:30:47.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.380+0000 7fc553ded780 1 < completed write operation key=8GRA2MUBGWJI tag=K4448DOXNRJG type=write size=648181 2023-12-03T11:30:47.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.382+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=PZETEA3QKSSF type=del size=2045757 2023-12-03T11:30:47.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.383+0000 7fc553ded780 1 < completed write operation key=UKAL8FCVIRSH tag=97OCPJ2QRJG6 type=write size=1015715 2023-12-03T11:30:47.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.385+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~MJ0U12G8FV92.0 size=428971 2023-12-03T11:30:47.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.385+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=MJ0U12G8FV92 tag=QZNPS2STP2G1 type=write size=1477547 2023-12-03T11:30:47.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.388+0000 7fc553ded780 1 < suggested update operation key=588S2ASN9I0W tag=DQSJZHHPGIZ8 type=del size=1499106 2023-12-03T11:30:47.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.389+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=NGZC4PELYS6B type=del size=4154169 2023-12-03T11:30:47.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.390+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=2ZVN9ICMSGQB type=del 2023-12-03T11:30:47.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.391+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=JD1LKXK9G8MZ type=del size=598294 2023-12-03T11:30:47.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.393+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=8KP0XIGKWVPO type=del 2023-12-03T11:30:47.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.394+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=FQ5SYAD99EI6 type=del size=3788595 2023-12-03T11:30:47.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.395+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~98WRP2N0A4C0.0 size=1048576 2023-12-03T11:30:47.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.396+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~98WRP2N0A4C0.1 size=209633 2023-12-03T11:30:47.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.398+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=98WRP2N0A4C0 tag=BRYA49YDM9MV type=write size=2306785 2023-12-03T11:30:47.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.401+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~EUFV7DUBRJ5A.0 size=1048576 2023-12-03T11:30:47.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.402+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~EUFV7DUBRJ5A.1 size=612684 2023-12-03T11:30:47.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.402+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=EUFV7DUBRJ5A tag=H1QM7IN9P56V type=write size=2709836 2023-12-03T11:30:47.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.403+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=B5V5RY7DSS1Z type=del 2023-12-03T11:30:47.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.404+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=AIVTEQYOCH22 type=del size=1699328 2023-12-03T11:30:47.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.406+0000 7fc553ded780 1 < canceled operation key=I3U08QX8MLBD tag=1J0YWNDPOGHQ type=del size=2002390 2023-12-03T11:30:47.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.407+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=O4RKYUCSS49N type=write size=268533 2023-12-03T11:30:47.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.409+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~0JMSS9ANLOMU.0 size=1048576 2023-12-03T11:30:47.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.409+0000 7fc553ded780 1 < canceled part key=_multipart_0H5OZMBMLQFJ.2~0JMSS9ANLOMU.1 size=425324 2023-12-03T11:30:47.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.411+0000 7fc553ded780 1 < canceled multipart upload key=0H5OZMBMLQFJ upload=0JMSS9ANLOMU tag=PBCU0Q2JJN4V type=write size=2522476 2023-12-03T11:30:47.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.412+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=RIB369KUHQCD type=del size=3038810 2023-12-03T11:30:47.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.413+0000 7fc553ded780 1 < canceled part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.0 size=1048576 2023-12-03T11:30:47.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.414+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.1 size=1048576 2023-12-03T11:30:47.416 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.415+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~1E93J39WJNHS.2 size=476729 2023-12-03T11:30:47.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.417+0000 7fc553ded780 1 < canceled multipart upload key=J664UUE697Q8 upload=1E93J39WJNHS tag=TISQYRMDQJJT type=write size=3622457 2023-12-03T11:30:47.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.419+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~DXZAFY1XHKVU.0 size=1048576 2023-12-03T11:30:47.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.421+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~DXZAFY1XHKVU.1 size=1032157 2023-12-03T11:30:47.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.421+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=DXZAFY1XHKVU tag=X4HEH4B3CJTN type=write size=3129309 2023-12-03T11:30:47.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.422+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~4XPXU61630XC.0 size=1048576 2023-12-03T11:30:47.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.423+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~4XPXU61630XC.1 size=260089 2023-12-03T11:30:47.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.426+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=4XPXU61630XC tag=QC32ABT1EUVS type=write size=2357241 2023-12-03T11:30:47.428 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.427+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=PUCVW004LNPZ type=del size=1015725 2023-12-03T11:30:47.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.430+0000 7fc553ded780 1 < suggested update operation key=ICDF1EYNI5IU tag=Z8CU3M68966C type=write size=737870 2023-12-03T11:30:47.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.432+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.0 size=1048576 2023-12-03T11:30:47.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.434+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.1 size=1048576 2023-12-03T11:30:47.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.434+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.2 size=798930 2023-12-03T11:30:47.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.434+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=Z1N48IIC20DS tag=J3FV0SL8D4OJ type=write size=3944658 2023-12-03T11:30:47.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.436+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=0PYDQCVJELX3 type=del 2023-12-03T11:30:47.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.437+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=NGHGIHRX2ABU type=write size=358053 2023-12-03T11:30:47.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.439+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=05QFD7EWRB2Q type=del 2023-12-03T11:30:47.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.441+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.0 size=1048576 2023-12-03T11:30:47.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.444+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.1 size=1048576 2023-12-03T11:30:47.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.445+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.2 size=184067 2023-12-03T11:30:47.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.445+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=QC5TLFF7FVXK tag=X74S97ZPL4LI type=write size=3329795 2023-12-03T11:30:47.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.446+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.0 size=1048576 2023-12-03T11:30:47.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.447+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.1 size=1048576 2023-12-03T11:30:47.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.447+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~M95SAWKC1AE1.2 size=445029 2023-12-03T11:30:47.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.449+0000 7fc553ded780 1 < canceled multipart upload key=EK7V6U9DSLD0 upload=M95SAWKC1AE1 tag=WE0QDJDFMG0D type=write size=3590757 2023-12-03T11:30:47.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.452+0000 7fc553ded780 1 > prepared part key=_multipart_UKAL8FCVIRSH.2~8W3SBVF0G3HH.0 size=203541 2023-12-03T11:30:47.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.452+0000 7fc553ded780 1 > prepared multipart upload key=UKAL8FCVIRSH upload=8W3SBVF0G3HH tag=X31ZOCSJOX9P type=write size=1252117 2023-12-03T11:30:47.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.453+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=4G1O9VED41Q1 type=del 2023-12-03T11:30:47.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.455+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.0 size=1048576 2023-12-03T11:30:47.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.456+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.1 size=1048576 2023-12-03T11:30:47.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.457+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.2 size=346660 2023-12-03T11:30:47.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.457+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=LCRJMYF2NO0U tag=ZEKLPWZT4AI8 type=write size=3492388 2023-12-03T11:30:47.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.458+0000 7fc553ded780 1 < canceled operation key=8GRA2MUBGWJI tag=PZETEA3QKSSF type=del size=2045757 2023-12-03T11:30:47.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.459+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=B4GRGJHGM870 type=del size=1389631 2023-12-03T11:30:47.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.461+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~MJ0U12G8FV92.0 size=428971 2023-12-03T11:30:47.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.463+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=MJ0U12G8FV92 tag=QZNPS2STP2G1 type=write size=1477547 2023-12-03T11:30:47.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.465+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~2B8XTJMU3UEV.0 size=150084 2023-12-03T11:30:47.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.465+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=2B8XTJMU3UEV tag=VRV1NB0INFPL type=write size=1198660 2023-12-03T11:30:47.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.466+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=NGZC4PELYS6B type=del 2023-12-03T11:30:47.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.469+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~8K06OQ8B1TZU.0 size=1048576 2023-12-03T11:30:47.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.470+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~8K06OQ8B1TZU.1 size=896803 2023-12-03T11:30:47.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.470+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=8K06OQ8B1TZU tag=0ZY204EQT29V type=write size=2993955 2023-12-03T11:30:47.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.471+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=JD1LKXK9G8MZ type=del 2023-12-03T11:30:47.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.472+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=84XVSLGQCU07 type=del size=3174490 2023-12-03T11:30:47.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.473+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=FQ5SYAD99EI6 type=del 2023-12-03T11:30:47.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.476+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~R9UID2U7OQC2.0 size=1048576 2023-12-03T11:30:47.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.477+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~R9UID2U7OQC2.1 size=51035 2023-12-03T11:30:47.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.477+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=R9UID2U7OQC2 tag=51TKD1D1K0KX type=write size=2148187 2023-12-03T11:30:47.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.478+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~EUFV7DUBRJ5A.0 size=1048576 2023-12-03T11:30:47.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.479+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~EUFV7DUBRJ5A.1 size=612684 2023-12-03T11:30:47.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.481+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=EUFV7DUBRJ5A tag=H1QM7IN9P56V type=write size=2709836 2023-12-03T11:30:47.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.482+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=W7JGDH665FR0 type=del size=3921033 2023-12-03T11:30:47.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.485+0000 7fc553ded780 1 < suggested remove operation key=I3U08QX8MLBD tag=AIVTEQYOCH22 type=del 2023-12-03T11:30:47.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.487+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~75CCB20EO6TI.0 size=757587 2023-12-03T11:30:47.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.487+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=75CCB20EO6TI tag=M6WB33U2T1MS type=write size=1806163 2023-12-03T11:30:47.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.488+0000 7fc553ded780 1 < completed write operation key=588S2ASN9I0W tag=O4RKYUCSS49N type=write size=268533 2023-12-03T11:30:47.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.489+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=SD8VHCYLW4YF type=del size=2971131 2023-12-03T11:30:47.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.491+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=RIB369KUHQCD type=del 2023-12-03T11:30:47.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.492+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=3DI67IF3WYRL type=del size=360082 2023-12-03T11:30:47.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.492+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~DXZAFY1XHKVU.0 size=1048576 2023-12-03T11:30:47.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.493+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~DXZAFY1XHKVU.1 size=1032157 2023-12-03T11:30:47.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.495+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=DXZAFY1XHKVU tag=X4HEH4B3CJTN type=write size=3129309 2023-12-03T11:30:47.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.498+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.0 size=1048576 2023-12-03T11:30:47.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.499+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.1 size=1048576 2023-12-03T11:30:47.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.500+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.2 size=938520 2023-12-03T11:30:47.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.500+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=ECFCC0UQCBS8 tag=RK63Y2K0Z4IV type=write size=4084248 2023-12-03T11:30:47.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.501+0000 7fc553ded780 1 < canceled operation key=0H5OZMBMLQFJ tag=PUCVW004LNPZ type=del size=1015725 2023-12-03T11:30:47.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.502+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=BGDRJCMBQFQN type=del size=85927 2023-12-03T11:30:47.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.504+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.0 size=1048576 2023-12-03T11:30:47.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.504+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.1 size=1048576 2023-12-03T11:30:47.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.505+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~Z1N48IIC20DS.2 size=798930 2023-12-03T11:30:47.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.507+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=Z1N48IIC20DS tag=J3FV0SL8D4OJ type=write size=3944658 2023-12-03T11:30:47.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.509+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.0 size=1048576 2023-12-03T11:30:47.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.510+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.1 size=1048576 2023-12-03T11:30:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.511+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.2 size=951246 2023-12-03T11:30:47.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.511+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=N68151GQX478 tag=Y5CBDB7LPVEB type=write size=4096974 2023-12-03T11:30:47.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.512+0000 7fc553ded780 1 < completed write operation key=IRMADF9MXXBZ tag=NGHGIHRX2ABU type=write size=358053 2023-12-03T11:30:47.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.513+0000 7fc553ded780 1 > prepared operation key=LRN5NYUZ5JT9 tag=PDJVZKM6L9S7 type=del size=1771418 2023-12-03T11:30:47.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.515+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.0 size=1048576 2023-12-03T11:30:47.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.516+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.1 size=1048576 2023-12-03T11:30:47.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.517+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~QC5TLFF7FVXK.2 size=184067 2023-12-03T11:30:47.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.519+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=QC5TLFF7FVXK tag=X74S97ZPL4LI type=write size=3329795 2023-12-03T11:30:47.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.520+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=8GOZQO52HX81 type=del size=119512 2023-12-03T11:30:47.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.522+0000 7fc553ded780 1 < completed part key=_multipart_UKAL8FCVIRSH.2~8W3SBVF0G3HH.0 size=203541 2023-12-03T11:30:47.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.524+0000 7fc553ded780 1 < completed multipart upload key=UKAL8FCVIRSH upload=8W3SBVF0G3HH tag=X31ZOCSJOX9P type=write size=1252117 2023-12-03T11:30:47.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.526+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=5VTM7TJVN5PH type=del size=3511386 2023-12-03T11:30:47.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.527+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.0 size=1048576 2023-12-03T11:30:47.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.529+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.1 size=1048576 2023-12-03T11:30:47.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.530+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~LCRJMYF2NO0U.2 size=346660 2023-12-03T11:30:47.533 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.532+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=LCRJMYF2NO0U tag=ZEKLPWZT4AI8 type=write size=3492388 2023-12-03T11:30:47.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.534+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=O0Y1GE42E9V6 type=del size=2486300 2023-12-03T11:30:47.536 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.535+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=B4GRGJHGM870 type=del 2023-12-03T11:30:47.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.536+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=UFK87RXOVAHT type=del size=2099125 2023-12-03T11:30:47.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.537+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~2B8XTJMU3UEV.0 size=150084 2023-12-03T11:30:47.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.539+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=2B8XTJMU3UEV tag=VRV1NB0INFPL type=write size=1198660 2023-12-03T11:30:47.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.541+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=MPESS53OBSAC type=del size=3986547 2023-12-03T11:30:47.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.542+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~8K06OQ8B1TZU.0 size=1048576 2023-12-03T11:30:47.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.543+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~8K06OQ8B1TZU.1 size=896803 2023-12-03T11:30:47.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.545+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=8K06OQ8B1TZU tag=0ZY204EQT29V type=write size=2993955 2023-12-03T11:30:47.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.546+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=8SCH2T00ER4S type=del size=3870757 2023-12-03T11:30:47.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.549+0000 7fc553ded780 1 < suggested remove operation key=4TWM1QM1OHOF tag=84XVSLGQCU07 type=del 2023-12-03T11:30:47.551 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.550+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=1URUZM0JI730 type=del size=1625258 2023-12-03T11:30:47.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.551+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~R9UID2U7OQC2.0 size=1048576 2023-12-03T11:30:47.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.552+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~R9UID2U7OQC2.1 size=51035 2023-12-03T11:30:47.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.554+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=R9UID2U7OQC2 tag=51TKD1D1K0KX type=write size=2148187 2023-12-03T11:30:47.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.557+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~40NWO2MXDOUC.0 size=17830 2023-12-03T11:30:47.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.557+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=40NWO2MXDOUC tag=ZDYZXXS3JG93 type=write size=1066406 2023-12-03T11:30:47.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.558+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=W7JGDH665FR0 type=del 2023-12-03T11:30:47.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.559+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=7JP3Y7Y6N0FM type=del size=1622149 2023-12-03T11:30:47.562 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.561+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~75CCB20EO6TI.0 size=757587 2023-12-03T11:30:47.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.563+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=75CCB20EO6TI tag=M6WB33U2T1MS type=write size=1806163 2023-12-03T11:30:47.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.564+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=0BNGBCU6QOQ3 type=write size=385529 2023-12-03T11:30:47.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.566+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=SD8VHCYLW4YF type=del 2023-12-03T11:30:47.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.567+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=FRE2YFB1J6LW type=del size=3533553 2023-12-03T11:30:47.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.568+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=3DI67IF3WYRL type=del 2023-12-03T11:30:47.572 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.571+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~W5ICCFC9KIWL.0 size=1048576 2023-12-03T11:30:47.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.572+0000 7fc553ded780 1 > prepared part key=_multipart_SWTRHXTAR0A1.2~W5ICCFC9KIWL.1 size=557951 2023-12-03T11:30:47.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.572+0000 7fc553ded780 1 > prepared multipart upload key=SWTRHXTAR0A1 upload=W5ICCFC9KIWL tag=DMTXVB3BE2HN type=write size=2655103 2023-12-03T11:30:47.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.573+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.0 size=1048576 2023-12-03T11:30:47.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.574+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.1 size=1048576 2023-12-03T11:30:47.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.574+0000 7fc553ded780 1 < canceled part key=_multipart_ICDF1EYNI5IU.2~ECFCC0UQCBS8.2 size=938520 2023-12-03T11:30:47.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.576+0000 7fc553ded780 1 < canceled multipart upload key=ICDF1EYNI5IU upload=ECFCC0UQCBS8 tag=RK63Y2K0Z4IV type=write size=4084248 2023-12-03T11:30:47.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.577+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=C5I99HYUAWNG type=del size=174633 2023-12-03T11:30:47.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.579+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=BGDRJCMBQFQN type=del 2023-12-03T11:30:47.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.581+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~W1UOP9EBSHUW.0 size=1048576 2023-12-03T11:30:47.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.582+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~W1UOP9EBSHUW.1 size=412434 2023-12-03T11:30:47.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.582+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=W1UOP9EBSHUW tag=SC88YNVWSR43 type=write size=2509586 2023-12-03T11:30:47.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.583+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.0 size=1048576 2023-12-03T11:30:47.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.584+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.1 size=1048576 2023-12-03T11:30:47.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.584+0000 7fc553ded780 1 < canceled part key=_multipart_PL22GPUFN3O0.2~N68151GQX478.2 size=951246 2023-12-03T11:30:47.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.586+0000 7fc553ded780 1 < canceled multipart upload key=PL22GPUFN3O0 upload=N68151GQX478 tag=Y5CBDB7LPVEB type=write size=4096974 2023-12-03T11:30:47.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.589+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.0 size=1048576 2023-12-03T11:30:47.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.590+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.1 size=1048576 2023-12-03T11:30:47.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.591+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.2 size=473129 2023-12-03T11:30:47.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.591+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=I14BGN7KR2IH tag=GQKUAJZ5F286 type=write size=3618857 2023-12-03T11:30:47.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.592+0000 7fc553ded780 1 < completed delete operation key=LRN5NYUZ5JT9 tag=PDJVZKM6L9S7 type=del 2023-12-03T11:30:47.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.595+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=AWKN8D0KCFF2 type=write size=520119 2023-12-03T11:30:47.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.596+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=8GOZQO52HX81 type=del size=119512 2023-12-03T11:30:47.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.598+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=M39UMUY58R0E type=del size=2703824 2023-12-03T11:30:47.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.599+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=5VTM7TJVN5PH type=del 2023-12-03T11:30:47.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.600+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=WJR2R6A94U7G type=write size=425232 2023-12-03T11:30:47.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.602+0000 7fc553ded780 1 < completed delete operation key=WLTMDGRBF4C9 tag=O0Y1GE42E9V6 type=del 2023-12-03T11:30:47.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.603+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=VRM8XELDU8CT type=del size=1789857 2023-12-03T11:30:47.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.605+0000 7fc553ded780 1 < suggested update operation key=J664UUE697Q8 tag=UFK87RXOVAHT type=del size=2099125 2023-12-03T11:30:47.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.607+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.0 size=1048576 2023-12-03T11:30:47.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.608+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.1 size=1048576 2023-12-03T11:30:47.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.609+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.2 size=87348 2023-12-03T11:30:47.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.609+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=ZOVZD7A53KD8 tag=PCMWHP7CJ027 type=write size=3233076 2023-12-03T11:30:47.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.611+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=MPESS53OBSAC type=del 2023-12-03T11:30:47.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.613+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~KR02PQLO66M8.0 size=1048576 2023-12-03T11:30:47.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.614+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~KR02PQLO66M8.1 size=54376 2023-12-03T11:30:47.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.614+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=KR02PQLO66M8 tag=PG1TRQY544WJ type=write size=2151528 2023-12-03T11:30:47.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.615+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=8SCH2T00ER4S type=del 2023-12-03T11:30:47.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.617+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~QETW7BJUAZKC.0 size=1048576 2023-12-03T11:30:47.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.618+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~QETW7BJUAZKC.1 size=150625 2023-12-03T11:30:47.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.618+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=QETW7BJUAZKC tag=7OZKLWOJMHEW type=write size=2247777 2023-12-03T11:30:47.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.620+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=1URUZM0JI730 type=del 2023-12-03T11:30:47.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.622+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~K69FA77WFINZ.0 size=1048576 2023-12-03T11:30:47.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.623+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~K69FA77WFINZ.1 size=129390 2023-12-03T11:30:47.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.623+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=K69FA77WFINZ tag=0WZGB18I80D4 type=write size=2226542 2023-12-03T11:30:47.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.624+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~40NWO2MXDOUC.0 size=17830 2023-12-03T11:30:47.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.626+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=40NWO2MXDOUC tag=ZDYZXXS3JG93 type=write size=1066406 2023-12-03T11:30:47.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.627+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=4TV1KIYSMLSH type=del size=3388017 2023-12-03T11:30:47.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.629+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=7JP3Y7Y6N0FM type=del 2023-12-03T11:30:47.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.630+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=FICCCRJKSISR type=del size=2715563 2023-12-03T11:30:47.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.632+0000 7fc553ded780 1 < suggested remove operation key=WDV4CW863V33 tag=0BNGBCU6QOQ3 type=write 2023-12-03T11:30:47.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.633+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=8Q7XFBYXJJOH type=del size=634111 2023-12-03T11:30:47.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.635+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=FRE2YFB1J6LW type=del 2023-12-03T11:30:47.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.636+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=KUU8B6TG20WN type=del size=253618 2023-12-03T11:30:47.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.637+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~W5ICCFC9KIWL.0 size=1048576 2023-12-03T11:30:47.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.638+0000 7fc553ded780 1 < completed part key=_multipart_SWTRHXTAR0A1.2~W5ICCFC9KIWL.1 size=557951 2023-12-03T11:30:47.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.640+0000 7fc553ded780 1 < completed multipart upload key=SWTRHXTAR0A1 upload=W5ICCFC9KIWL tag=DMTXVB3BE2HN type=write size=2655103 2023-12-03T11:30:47.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.641+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=C18X8P67GV0B type=del size=968040 2023-12-03T11:30:47.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.643+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=C5I99HYUAWNG type=del 2023-12-03T11:30:47.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.644+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=RLSPB0046UXA type=del size=631719 2023-12-03T11:30:47.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.645+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~W1UOP9EBSHUW.0 size=1048576 2023-12-03T11:30:47.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.646+0000 7fc553ded780 1 < completed part key=_multipart_C2TPF9Q5Z7XQ.2~W1UOP9EBSHUW.1 size=412434 2023-12-03T11:30:47.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.648+0000 7fc553ded780 1 < completed multipart upload key=C2TPF9Q5Z7XQ upload=W1UOP9EBSHUW tag=SC88YNVWSR43 type=write size=2509586 2023-12-03T11:30:47.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.650+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.0 size=1048576 2023-12-03T11:30:47.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.651+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.1 size=1048576 2023-12-03T11:30:47.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.652+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.2 size=325958 2023-12-03T11:30:47.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.652+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=NWWPRPIA1P8U tag=JKOL3F2KVUV3 type=write size=3471686 2023-12-03T11:30:47.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.654+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.0 size=1048576 2023-12-03T11:30:47.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.655+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.1 size=1048576 2023-12-03T11:30:47.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.656+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~I14BGN7KR2IH.2 size=473129 2023-12-03T11:30:47.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.658+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=I14BGN7KR2IH tag=GQKUAJZ5F286 type=write size=3618857 2023-12-03T11:30:47.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.659+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=0GPPMOIN72E7 type=del size=1517679 2023-12-03T11:30:47.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.661+0000 7fc553ded780 1 < completed write operation key=JUYA4LXEQ4US tag=AWKN8D0KCFF2 type=write size=520119 2023-12-03T11:30:47.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.662+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=47LDTI6ZSH8H type=del size=3365685 2023-12-03T11:30:47.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.664+0000 7fc553ded780 1 < suggested remove operation key=EK7V6U9DSLD0 tag=M39UMUY58R0E type=del 2023-12-03T11:30:47.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.665+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=7M5N2JRSW873 type=write size=364919 2023-12-03T11:30:47.668 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.667+0000 7fc553ded780 1 < completed write operation key=WOKG1DISIKK0 tag=WJR2R6A94U7G type=write size=425232 2023-12-03T11:30:47.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.668+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=LELB2UC8LGEX type=del size=3943855 2023-12-03T11:30:47.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.669+0000 7fc553ded780 -1 < failed to complete operation key=WDV4CW863V33 tag=VRM8XELDU8CT type=del size=1789857: (22) Invalid argument 2023-12-03T11:30:47.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.672+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~T8TB3SBXEBCP.0 size=642199 2023-12-03T11:30:47.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.672+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=T8TB3SBXEBCP tag=N24BKWJ414OL type=write size=1690775 2023-12-03T11:30:47.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.673+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.0 size=1048576 2023-12-03T11:30:47.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.674+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.1 size=1048576 2023-12-03T11:30:47.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.675+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~ZOVZD7A53KD8.2 size=87348 2023-12-03T11:30:47.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.678+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=ZOVZD7A53KD8 tag=PCMWHP7CJ027 type=write size=3233076 2023-12-03T11:30:47.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.682+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~CU60XDOR7PN5.0 size=1048576 2023-12-03T11:30:47.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.683+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~CU60XDOR7PN5.1 size=890335 2023-12-03T11:30:47.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.683+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=CU60XDOR7PN5 tag=MV2N176EEDZM type=write size=2987487 2023-12-03T11:30:47.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.684+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~KR02PQLO66M8.0 size=1048576 2023-12-03T11:30:47.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.685+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~KR02PQLO66M8.1 size=54376 2023-12-03T11:30:47.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.688+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=KR02PQLO66M8 tag=PG1TRQY544WJ type=write size=2151528 2023-12-03T11:30:47.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.690+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~OTBLOOGIN1NC.0 size=1048576 2023-12-03T11:30:47.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.691+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~OTBLOOGIN1NC.1 size=361280 2023-12-03T11:30:47.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.691+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=OTBLOOGIN1NC tag=L0PWB8FCDA5K type=write size=2458432 2023-12-03T11:30:47.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.693+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~QETW7BJUAZKC.0 size=1048576 2023-12-03T11:30:47.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.694+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~QETW7BJUAZKC.1 size=150625 2023-12-03T11:30:47.697 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.696+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=QETW7BJUAZKC tag=7OZKLWOJMHEW type=write size=2247777 2023-12-03T11:30:47.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.698+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~L38KLQOWXDIA.0 size=63040 2023-12-03T11:30:47.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.698+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=L38KLQOWXDIA tag=9CF1UE7PJRMS type=write size=1111616 2023-12-03T11:30:47.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.699+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~K69FA77WFINZ.0 size=1048576 2023-12-03T11:30:47.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.700+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~K69FA77WFINZ.1 size=129390 2023-12-03T11:30:47.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.702+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=K69FA77WFINZ tag=0WZGB18I80D4 type=write size=2226542 2023-12-03T11:30:47.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.705+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.0 size=1048576 2023-12-03T11:30:47.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.706+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.1 size=1048576 2023-12-03T11:30:47.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.707+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.2 size=846142 2023-12-03T11:30:47.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.707+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=K2A321MSZVBZ tag=N4RVO192PSRX type=write size=3991870 2023-12-03T11:30:47.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.708+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=4TV1KIYSMLSH type=del 2023-12-03T11:30:47.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.711+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=EENYSL9R1VRA type=del size=570289 2023-12-03T11:30:47.713 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.712+0000 7fc553ded780 -1 < failed to complete operation key=WDV4CW863V33 tag=FICCCRJKSISR type=del size=2715563: (22) Invalid argument 2023-12-03T11:30:47.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.713+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=AFNQNCO3IOX5 type=del size=2624289 2023-12-03T11:30:47.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.716+0000 7fc553ded780 1 < suggested remove operation key=4TWM1QM1OHOF tag=8Q7XFBYXJJOH type=del 2023-12-03T11:30:47.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.718+0000 7fc553ded780 1 > prepared part key=_multipart_J664UUE697Q8.2~8WGV3IUQB3MS.0 size=890576 2023-12-03T11:30:47.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.718+0000 7fc553ded780 1 > prepared multipart upload key=J664UUE697Q8 upload=8WGV3IUQB3MS tag=G28F47QZPDHN type=write size=1939152 2023-12-03T11:30:47.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.720+0000 7fc553ded780 1 < suggested update operation key=X741JIRT7RIG tag=KUU8B6TG20WN type=del size=253618 2023-12-03T11:30:47.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.723+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.0 size=1048576 2023-12-03T11:30:47.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.724+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.1 size=1048576 2023-12-03T11:30:47.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.724+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.2 size=649916 2023-12-03T11:30:47.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.724+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=B2TMFOYVDFJ8 tag=ZC4BPZY80IMU type=write size=3795644 2023-12-03T11:30:47.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.726+0000 7fc553ded780 1 < completed delete operation key=ICDF1EYNI5IU tag=C18X8P67GV0B type=del 2023-12-03T11:30:47.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.728+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.0 size=1048576 2023-12-03T11:30:47.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.729+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.1 size=1048576 2023-12-03T11:30:47.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.730+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.2 size=253927 2023-12-03T11:30:47.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.730+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=NKDDPMEL31MX tag=5G37D0KAJXN8 type=write size=3399655 2023-12-03T11:30:47.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.733+0000 7fc553ded780 1 < suggested update operation key=I3U08QX8MLBD tag=RLSPB0046UXA type=del size=631719 2023-12-03T11:30:47.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.736+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~COSG4GPKH6K5.0 size=1021569 2023-12-03T11:30:47.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.736+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=COSG4GPKH6K5 tag=P3XPRLXP7RGS type=write size=2070145 2023-12-03T11:30:47.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.736+0000 7fc553ded780 1 < canceled part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.0 size=1048576 2023-12-03T11:30:47.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.736+0000 7fc553ded780 1 < canceled part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.1 size=1048576 2023-12-03T11:30:47.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.737+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~NWWPRPIA1P8U.2 size=325958 2023-12-03T11:30:47.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.740+0000 7fc553ded780 1 < canceled multipart upload key=FC4WU6SMNEWV upload=NWWPRPIA1P8U tag=JKOL3F2KVUV3 type=write size=3471686 2023-12-03T11:30:47.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.741+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=Y0V25VGWIYCD type=del size=2716074 2023-12-03T11:30:47.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.742+0000 7fc553ded780 1 < completed delete operation key=JUYA4LXEQ4US tag=0GPPMOIN72E7 type=del 2023-12-03T11:30:47.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.744+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=7DCHMBTULXYA type=del size=3384600 2023-12-03T11:30:47.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.745+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=47LDTI6ZSH8H type=del 2023-12-03T11:30:47.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.747+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.0 size=1048576 2023-12-03T11:30:47.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.748+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.1 size=1048576 2023-12-03T11:30:47.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.749+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.2 size=508130 2023-12-03T11:30:47.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.749+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=MQ2FAQ1M38CW tag=9X9Z7F4DI4Q8 type=write size=3653858 2023-12-03T11:30:47.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.751+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=7M5N2JRSW873 type=write size=364919 2023-12-03T11:30:47.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.752+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=4SH2VE2A0F1T type=del size=3372845 2023-12-03T11:30:47.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.753+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=LELB2UC8LGEX type=del 2023-12-03T11:30:47.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.754+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=XHA1J1ZVX2VM type=del size=1333735 2023-12-03T11:30:47.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.756+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~T8TB3SBXEBCP.0 size=642199 2023-12-03T11:30:47.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.758+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=T8TB3SBXEBCP tag=N24BKWJ414OL type=write size=1690775 2023-12-03T11:30:47.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.759+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=E8ADXXDFF6KJ type=del size=1152799 2023-12-03T11:30:47.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.760+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~CU60XDOR7PN5.0 size=1048576 2023-12-03T11:30:47.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.761+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~CU60XDOR7PN5.1 size=890335 2023-12-03T11:30:47.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.763+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=CU60XDOR7PN5 tag=MV2N176EEDZM type=write size=2987487 2023-12-03T11:30:47.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.764+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=J49ZKSGFCV61 type=del size=1504577 2023-12-03T11:30:47.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.766+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~OTBLOOGIN1NC.0 size=1048576 2023-12-03T11:30:47.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.767+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~OTBLOOGIN1NC.1 size=361280 2023-12-03T11:30:47.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.769+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=OTBLOOGIN1NC tag=L0PWB8FCDA5K type=write size=2458432 2023-12-03T11:30:47.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.770+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=D8FQ9B5DCCUT type=del size=3798542 2023-12-03T11:30:47.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.771+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~L38KLQOWXDIA.0 size=63040 2023-12-03T11:30:47.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.773+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=L38KLQOWXDIA tag=9CF1UE7PJRMS type=write size=1111616 2023-12-03T11:30:47.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.776+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~OQXHFDYJ56UV.0 size=658641 2023-12-03T11:30:47.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.776+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=OQXHFDYJ56UV tag=ZF4GC0HL8HKM type=write size=1707217 2023-12-03T11:30:47.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.777+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.0 size=1048576 2023-12-03T11:30:47.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.778+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.1 size=1048576 2023-12-03T11:30:47.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.779+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~K2A321MSZVBZ.2 size=846142 2023-12-03T11:30:47.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.781+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=K2A321MSZVBZ tag=N4RVO192PSRX type=write size=3991870 2023-12-03T11:30:47.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.783+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.0 size=1048576 2023-12-03T11:30:47.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.784+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.1 size=1048576 2023-12-03T11:30:47.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.785+0000 7fc553ded780 1 > prepared part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.2 size=422381 2023-12-03T11:30:47.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.785+0000 7fc553ded780 1 > prepared multipart upload key=ZUNOFY5N3J3H upload=IK8ZZ8PUOBO8 tag=VS6STKEE978E type=write size=3568109 2023-12-03T11:30:47.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.786+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=EENYSL9R1VRA type=del 2023-12-03T11:30:47.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.787+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=GRS61YJIN4WZ type=write size=627946 2023-12-03T11:30:47.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.789+0000 7fc553ded780 -1 < failed to complete operation key=4TWM1QM1OHOF tag=AFNQNCO3IOX5 type=del size=2624289: (22) Invalid argument 2023-12-03T11:30:47.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.790+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=HTEQ64H70VQR type=del size=3636615 2023-12-03T11:30:47.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.791+0000 7fc553ded780 1 < completed part key=_multipart_J664UUE697Q8.2~8WGV3IUQB3MS.0 size=890576 2023-12-03T11:30:47.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.793+0000 7fc553ded780 1 < completed multipart upload key=J664UUE697Q8 upload=8WGV3IUQB3MS tag=G28F47QZPDHN type=write size=1939152 2023-12-03T11:30:47.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.795+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~D8BRIX3REVZZ.0 size=536142 2023-12-03T11:30:47.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.795+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=D8BRIX3REVZZ tag=BBS18HJ75LJ4 type=write size=1584718 2023-12-03T11:30:47.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.796+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.0 size=1048576 2023-12-03T11:30:47.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.796+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.1 size=1048576 2023-12-03T11:30:47.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.796+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~B2TMFOYVDFJ8.2 size=649916 2023-12-03T11:30:47.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.800+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=B2TMFOYVDFJ8 tag=ZC4BPZY80IMU type=write size=3795644 2023-12-03T11:30:47.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.802+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.0 size=1048576 2023-12-03T11:30:47.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.803+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.1 size=1048576 2023-12-03T11:30:47.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.804+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.2 size=896035 2023-12-03T11:30:47.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.804+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=LU462685G1UJ tag=SDOTXBMBG3PL type=write size=4041763 2023-12-03T11:30:47.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.805+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.0 size=1048576 2023-12-03T11:30:47.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.806+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.1 size=1048576 2023-12-03T11:30:47.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.807+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~NKDDPMEL31MX.2 size=253927 2023-12-03T11:30:47.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.809+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=NKDDPMEL31MX tag=5G37D0KAJXN8 type=write size=3399655 2023-12-03T11:30:47.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.812+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~65DHB7OR409X.0 size=1048576 2023-12-03T11:30:47.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.813+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~65DHB7OR409X.1 size=20494 2023-12-03T11:30:47.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.813+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=65DHB7OR409X tag=3AXK5QD5YK90 type=write size=2117646 2023-12-03T11:30:47.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.813+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~COSG4GPKH6K5.0 size=1021569 2023-12-03T11:30:47.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.815+0000 7fc553ded780 1 < canceled multipart upload key=EK7V6U9DSLD0 upload=COSG4GPKH6K5 tag=P3XPRLXP7RGS type=write size=2070145 2023-12-03T11:30:47.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.816+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=J9AHW9T33232 type=write size=703296 2023-12-03T11:30:47.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.817+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=Y0V25VGWIYCD type=del 2023-12-03T11:30:47.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.819+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=5A7QY3NEORKC type=del size=1153516 2023-12-03T11:30:47.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.821+0000 7fc553ded780 1 < suggested remove operation key=X741JIRT7RIG tag=7DCHMBTULXYA type=del 2023-12-03T11:30:47.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.823+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.0 size=1048576 2023-12-03T11:30:47.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.824+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.1 size=1048576 2023-12-03T11:30:47.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.825+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.2 size=717734 2023-12-03T11:30:47.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.825+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=I8SIU4D8RSLQ tag=579GPX4XC0QF type=write size=3863462 2023-12-03T11:30:47.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.825+0000 7fc553ded780 1 < canceled part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.0 size=1048576 2023-12-03T11:30:47.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.826+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.1 size=1048576 2023-12-03T11:30:47.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.827+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~MQ2FAQ1M38CW.2 size=508130 2023-12-03T11:30:47.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.829+0000 7fc553ded780 1 < canceled multipart upload key=EK7V6U9DSLD0 upload=MQ2FAQ1M38CW tag=9X9Z7F4DI4Q8 type=write size=3653858 2023-12-03T11:30:47.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.831+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=CYZ995D90XJA type=del size=3453750 2023-12-03T11:30:47.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.832+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=4SH2VE2A0F1T type=del 2023-12-03T11:30:47.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.834+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~JRKHHT3QNSKK.0 size=113969 2023-12-03T11:30:47.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.834+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=JRKHHT3QNSKK tag=V2E2ZJZCR1UG type=write size=1162545 2023-12-03T11:30:47.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.835+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=XHA1J1ZVX2VM type=del 2023-12-03T11:30:47.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.837+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=PH9P3NWC2T4X type=del size=1275120 2023-12-03T11:30:47.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.838+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=E8ADXXDFF6KJ type=del 2023-12-03T11:30:47.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.840+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.0 size=1048576 2023-12-03T11:30:47.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.841+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.1 size=1048576 2023-12-03T11:30:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.842+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.2 size=783776 2023-12-03T11:30:47.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.842+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=R3U2S1741W8E tag=RP7N9095UP0K type=write size=3929504 2023-12-03T11:30:47.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.844+0000 7fc553ded780 1 < suggested update operation key=6OW429S51UWN tag=J49ZKSGFCV61 type=del size=1504577 2023-12-03T11:30:47.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.845+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=1BHTP0G7J7JU type=write size=184428 2023-12-03T11:30:47.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.847+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=D8FQ9B5DCCUT type=del 2023-12-03T11:30:47.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.848+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=JXLFW5O9593N type=del size=3310832 2023-12-03T11:30:47.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.849+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~OQXHFDYJ56UV.0 size=658641 2023-12-03T11:30:47.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.851+0000 7fc553ded780 1 < completed multipart upload key=G1OBBD75M5ZQ upload=OQXHFDYJ56UV tag=ZF4GC0HL8HKM type=write size=1707217 2023-12-03T11:30:47.853 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.852+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=6QJ7VE7LLLYA type=del size=3582247 2023-12-03T11:30:47.854 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.853+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.0 size=1048576 2023-12-03T11:30:47.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.855+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.1 size=1048576 2023-12-03T11:30:47.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.855+0000 7fc553ded780 1 < completed part key=_multipart_ZUNOFY5N3J3H.2~IK8ZZ8PUOBO8.2 size=422381 2023-12-03T11:30:47.859 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.858+0000 7fc553ded780 1 < completed multipart upload key=ZUNOFY5N3J3H upload=IK8ZZ8PUOBO8 tag=VS6STKEE978E type=write size=3568109 2023-12-03T11:30:47.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.859+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=HU6D5C9XM9MP type=del size=171490 2023-12-03T11:30:47.861 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.860+0000 7fc553ded780 1 < completed write operation key=I3U08QX8MLBD tag=GRS61YJIN4WZ type=write size=627946 2023-12-03T11:30:47.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.862+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.0 size=1048576 2023-12-03T11:30:47.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.863+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.1 size=1048576 2023-12-03T11:30:47.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.864+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.2 size=441710 2023-12-03T11:30:47.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.864+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=0MJCBYPR51QL tag=PQIMXV1VYD36 type=write size=3587438 2023-12-03T11:30:47.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.866+0000 7fc553ded780 1 < suggested update operation key=8GRA2MUBGWJI tag=HTEQ64H70VQR type=del size=3636615 2023-12-03T11:30:47.870 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.869+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~82TU9TB3UDTO.0 size=1048576 2023-12-03T11:30:47.870 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.870+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~82TU9TB3UDTO.1 size=841146 2023-12-03T11:30:47.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.870+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=82TU9TB3UDTO tag=YFPPTBQDJTAV type=write size=2938298 2023-12-03T11:30:47.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.870+0000 7fc553ded780 1 < canceled part key=_multipart_TQRZT37AB9WC.2~D8BRIX3REVZZ.0 size=536142 2023-12-03T11:30:47.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.872+0000 7fc553ded780 1 < canceled multipart upload key=TQRZT37AB9WC upload=D8BRIX3REVZZ tag=BBS18HJ75LJ4 type=write size=1584718 2023-12-03T11:30:47.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.874+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~WQ2N0X81DT8R.0 size=1048576 2023-12-03T11:30:47.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.875+0000 7fc553ded780 1 > prepared part key=_multipart_CZP8OQRU641T.2~WQ2N0X81DT8R.1 size=875790 2023-12-03T11:30:47.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.875+0000 7fc553ded780 1 > prepared multipart upload key=CZP8OQRU641T upload=WQ2N0X81DT8R tag=6WMYVJY01LA7 type=write size=2972942 2023-12-03T11:30:47.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.876+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.0 size=1048576 2023-12-03T11:30:47.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.877+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.1 size=1048576 2023-12-03T11:30:47.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.878+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~LU462685G1UJ.2 size=896035 2023-12-03T11:30:47.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.880+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=LU462685G1UJ tag=SDOTXBMBG3PL type=write size=4041763 2023-12-03T11:30:47.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.881+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=RVG24WUU7WPY type=del size=818085 2023-12-03T11:30:47.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.883+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~65DHB7OR409X.0 size=1048576 2023-12-03T11:30:47.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.884+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~65DHB7OR409X.1 size=20494 2023-12-03T11:30:47.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.886+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=65DHB7OR409X tag=3AXK5QD5YK90 type=write size=2117646 2023-12-03T11:30:47.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.887+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=1H64UV8CVPYZ type=del size=3201707 2023-12-03T11:30:47.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.888+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=J9AHW9T33232 type=write size=703296 2023-12-03T11:30:47.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.890+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=9N40K7A9MN8O type=del size=2454160 2023-12-03T11:30:47.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.891+0000 7fc553ded780 1 < completed delete operation key=FC4WU6SMNEWV tag=5A7QY3NEORKC type=del 2023-12-03T11:30:47.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.892+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=A53F7CGUW7B6 type=del size=2458161 2023-12-03T11:30:47.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.893+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.0 size=1048576 2023-12-03T11:30:47.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.894+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.1 size=1048576 2023-12-03T11:30:47.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.895+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~I8SIU4D8RSLQ.2 size=717734 2023-12-03T11:30:47.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.897+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=I8SIU4D8RSLQ tag=579GPX4XC0QF type=write size=3863462 2023-12-03T11:30:47.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.900+0000 7fc553ded780 1 > prepared part key=_multipart_ICDF1EYNI5IU.2~C988D4NZPYGB.0 size=644834 2023-12-03T11:30:47.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.900+0000 7fc553ded780 1 > prepared multipart upload key=ICDF1EYNI5IU upload=C988D4NZPYGB tag=1OOS6RMX408S type=write size=1693410 2023-12-03T11:30:47.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.901+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=CYZ995D90XJA type=del 2023-12-03T11:30:47.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.902+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=TR599SO0UYI1 type=del size=2795792 2023-12-03T11:30:47.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.904+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~JRKHHT3QNSKK.0 size=113969 2023-12-03T11:30:47.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.908+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=JRKHHT3QNSKK tag=V2E2ZJZCR1UG type=write size=1162545 2023-12-03T11:30:47.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.911+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~0EXDJYV49679.0 size=467637 2023-12-03T11:30:47.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.911+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=0EXDJYV49679 tag=0AHJOLA1YI8A type=write size=1516213 2023-12-03T11:30:47.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.912+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=PH9P3NWC2T4X type=del 2023-12-03T11:30:47.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.914+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=9G3AYQG33T5V type=del size=2697931 2023-12-03T11:30:47.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.916+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.0 size=1048576 2023-12-03T11:30:47.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.917+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.1 size=1048576 2023-12-03T11:30:47.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.917+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~R3U2S1741W8E.2 size=783776 2023-12-03T11:30:47.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.920+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=R3U2S1741W8E tag=RP7N9095UP0K type=write size=3929504 2023-12-03T11:30:47.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.921+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=3WQ50WTAQ7NE type=del size=3696913 2023-12-03T11:30:47.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.922+0000 7fc553ded780 1 < completed write operation key=IRMADF9MXXBZ tag=1BHTP0G7J7JU type=write size=184428 2023-12-03T11:30:47.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.924+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.0 size=1048576 2023-12-03T11:30:47.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.925+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.1 size=1048576 2023-12-03T11:30:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.926+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.2 size=972779 2023-12-03T11:30:47.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.926+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=IAHJECE93ZGQ tag=VLD6CM8VZ65M type=write size=4118507 2023-12-03T11:30:47.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.927+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=JXLFW5O9593N type=del size=3310832 2023-12-03T11:30:47.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.929+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=4VAX20D10RR9 type=del size=2078590 2023-12-03T11:30:47.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.930+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=6QJ7VE7LLLYA type=del 2023-12-03T11:30:47.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.932+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~ZHM7UHMYJPPE.0 size=979913 2023-12-03T11:30:47.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.932+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=ZHM7UHMYJPPE tag=JA84JL462ETG type=write size=2028489 2023-12-03T11:30:47.934 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.933+0000 7fc553ded780 1 < completed delete operation key=YE73HBF64CS7 tag=HU6D5C9XM9MP type=del 2023-12-03T11:30:47.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.935+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.0 size=1048576 2023-12-03T11:30:47.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.936+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.1 size=1048576 2023-12-03T11:30:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.937+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.2 size=905124 2023-12-03T11:30:47.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.937+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=OM5IH28VBIQG tag=YKUCO7DKXQQZ type=write size=4050852 2023-12-03T11:30:47.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.938+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.0 size=1048576 2023-12-03T11:30:47.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.938+0000 7fc553ded780 1 < canceled part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.1 size=1048576 2023-12-03T11:30:47.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.939+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~0MJCBYPR51QL.2 size=441710 2023-12-03T11:30:47.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.941+0000 7fc553ded780 1 < canceled multipart upload key=RO0KGI2QMA71 upload=0MJCBYPR51QL tag=PQIMXV1VYD36 type=write size=3587438 2023-12-03T11:30:47.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.943+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=8R16AQGQDE4R type=del size=4075347 2023-12-03T11:30:47.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.944+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~82TU9TB3UDTO.0 size=1048576 2023-12-03T11:30:47.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.945+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~82TU9TB3UDTO.1 size=841146 2023-12-03T11:30:47.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.947+0000 7fc553ded780 1 < completed multipart upload key=WOKG1DISIKK0 upload=82TU9TB3UDTO tag=YFPPTBQDJTAV type=write size=2938298 2023-12-03T11:30:47.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.949+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~Y1GK4EL64F9B.0 size=527319 2023-12-03T11:30:47.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.949+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=Y1GK4EL64F9B tag=W70QXDJLOR95 type=write size=1575895 2023-12-03T11:30:47.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.950+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~WQ2N0X81DT8R.0 size=1048576 2023-12-03T11:30:47.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.951+0000 7fc553ded780 1 < completed part key=_multipart_CZP8OQRU641T.2~WQ2N0X81DT8R.1 size=875790 2023-12-03T11:30:47.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.954+0000 7fc553ded780 1 < completed multipart upload key=CZP8OQRU641T upload=WQ2N0X81DT8R tag=6WMYVJY01LA7 type=write size=2972942 2023-12-03T11:30:47.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.956+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~FPRIAAKVNP3W.0 size=541009 2023-12-03T11:30:47.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.956+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=FPRIAAKVNP3W tag=NWGVG25LCIVX type=write size=1589585 2023-12-03T11:30:47.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.958+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=RVG24WUU7WPY type=del size=818085 2023-12-03T11:30:47.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.960+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~W0IAO01A8EDP.0 size=1048576 2023-12-03T11:30:47.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.961+0000 7fc553ded780 1 > prepared part key=_multipart_PL22GPUFN3O0.2~W0IAO01A8EDP.1 size=777830 2023-12-03T11:30:47.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.961+0000 7fc553ded780 1 > prepared multipart upload key=PL22GPUFN3O0 upload=W0IAO01A8EDP tag=5IE9NUBJ04FC type=write size=2874982 2023-12-03T11:30:47.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.962+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=1H64UV8CVPYZ type=del 2023-12-03T11:30:47.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.963+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=ZMNFN1HLOZ7B type=write size=185113 2023-12-03T11:30:47.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.965+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=9N40K7A9MN8O type=del 2023-12-03T11:30:47.967 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.966+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=HSANIJVNRS01 type=del size=988238 2023-12-03T11:30:47.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.968+0000 7fc553ded780 1 < suggested remove operation key=PL22GPUFN3O0 tag=A53F7CGUW7B6 type=del 2023-12-03T11:30:47.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.969+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=0V6O1TQZHJFV type=del size=2931329 2023-12-03T11:30:47.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.970+0000 7fc553ded780 1 < completed part key=_multipart_ICDF1EYNI5IU.2~C988D4NZPYGB.0 size=644834 2023-12-03T11:30:47.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.972+0000 7fc553ded780 1 < completed multipart upload key=ICDF1EYNI5IU upload=C988D4NZPYGB tag=1OOS6RMX408S type=write size=1693410 2023-12-03T11:30:47.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.973+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=HIK85GNUY3XI type=write size=859 2023-12-03T11:30:47.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.974+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=TR599SO0UYI1 type=del 2023-12-03T11:30:47.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.976+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=S5YO0FNCTD9O type=del size=889398 2023-12-03T11:30:47.978 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.977+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~0EXDJYV49679.0 size=467637 2023-12-03T11:30:47.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.979+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=0EXDJYV49679 tag=0AHJOLA1YI8A type=write size=1516213 2023-12-03T11:30:47.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.981+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.0 size=1048576 2023-12-03T11:30:47.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.982+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.1 size=1048576 2023-12-03T11:30:47.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.983+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.2 size=494253 2023-12-03T11:30:47.984 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.983+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=9DDXS3NZXR3I tag=E8U4YOF0BFHH type=write size=3639981 2023-12-03T11:30:47.986 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.985+0000 7fc553ded780 1 < suggested remove operation key=FC4WU6SMNEWV tag=9G3AYQG33T5V type=del 2023-12-03T11:30:47.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.987+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.0 size=1048576 2023-12-03T11:30:47.989 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.988+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.1 size=1048576 2023-12-03T11:30:47.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.989+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.2 size=714927 2023-12-03T11:30:47.990 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.989+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=22V94M6VB2DH tag=YFYPEN5S6F9E type=write size=3860655 2023-12-03T11:30:47.991 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.990+0000 7fc553ded780 1 < completed delete operation key=WOKG1DISIKK0 tag=3WQ50WTAQ7NE type=del 2023-12-03T11:30:47.992 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.991+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=IQHHUC3XG7KG type=del size=850297 2023-12-03T11:30:47.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.993+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.0 size=1048576 2023-12-03T11:30:47.995 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.994+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.1 size=1048576 2023-12-03T11:30:47.996 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.995+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~IAHJECE93ZGQ.2 size=972779 2023-12-03T11:30:47.998 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.997+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=IAHJECE93ZGQ tag=VLD6CM8VZ65M type=write size=4118507 2023-12-03T11:30:48.000 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:47.999+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~DZUWRZG10900.0 size=1048576 2023-12-03T11:30:48.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.000+0000 7fc553ded780 1 > prepared part key=_multipart_G1OBBD75M5ZQ.2~DZUWRZG10900.1 size=831436 2023-12-03T11:30:48.001 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.000+0000 7fc553ded780 1 > prepared multipart upload key=G1OBBD75M5ZQ upload=DZUWRZG10900 tag=Z1RV3K7V8SVO type=write size=2928588 2023-12-03T11:30:48.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.002+0000 7fc553ded780 1 < suggested remove operation key=G1OBBD75M5ZQ tag=4VAX20D10RR9 type=del 2023-12-03T11:30:48.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.004+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=UF6UN6YJFNJ0 type=del size=732857 2023-12-03T11:30:48.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.005+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~ZHM7UHMYJPPE.0 size=979913 2023-12-03T11:30:48.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.007+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=ZHM7UHMYJPPE tag=JA84JL462ETG type=write size=2028489 2023-12-03T11:30:48.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.009+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.0 size=1048576 2023-12-03T11:30:48.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.010+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.1 size=1048576 2023-12-03T11:30:48.011 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.010+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.2 size=649466 2023-12-03T11:30:48.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.011+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=KSK3HJNTCKVO tag=2VE4U5HRRY7Z type=write size=3795194 2023-12-03T11:30:48.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.011+0000 7fc553ded780 1 < canceled part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.0 size=1048576 2023-12-03T11:30:48.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.012+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.1 size=1048576 2023-12-03T11:30:48.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.013+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~OM5IH28VBIQG.2 size=905124 2023-12-03T11:30:48.016 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.015+0000 7fc553ded780 1 < canceled multipart upload key=X741JIRT7RIG upload=OM5IH28VBIQG tag=YKUCO7DKXQQZ type=write size=4050852 2023-12-03T11:30:48.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.017+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.0 size=1048576 2023-12-03T11:30:48.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.019+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.1 size=1048576 2023-12-03T11:30:48.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.019+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.2 size=941511 2023-12-03T11:30:48.020 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.019+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=1U9QMDGTGKDF tag=32MNNDFWFAPX type=write size=4087239 2023-12-03T11:30:48.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.021+0000 7fc553ded780 1 < completed delete operation key=WDV4CW863V33 tag=8R16AQGQDE4R type=del 2023-12-03T11:30:48.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.025+0000 7fc553ded780 1 > prepared part key=_multipart_X741JIRT7RIG.2~57EVTPQB33P8.0 size=948658 2023-12-03T11:30:48.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.025+0000 7fc553ded780 1 > prepared multipart upload key=X741JIRT7RIG upload=57EVTPQB33P8 tag=TRCYN42AXVKZ type=write size=1997234 2023-12-03T11:30:48.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.026+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~Y1GK4EL64F9B.0 size=527319 2023-12-03T11:30:48.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.029+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=Y1GK4EL64F9B tag=W70QXDJLOR95 type=write size=1575895 2023-12-03T11:30:48.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.031+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~0TEE365XUPP7.0 size=1048576 2023-12-03T11:30:48.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.032+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~0TEE365XUPP7.1 size=871903 2023-12-03T11:30:48.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.032+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=0TEE365XUPP7 tag=QKR2X2EP778K type=write size=2969055 2023-12-03T11:30:48.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.032+0000 7fc553ded780 1 < canceled part key=_multipart_LRN5NYUZ5JT9.2~FPRIAAKVNP3W.0 size=541009 2023-12-03T11:30:48.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.034+0000 7fc553ded780 1 < canceled multipart upload key=LRN5NYUZ5JT9 upload=FPRIAAKVNP3W tag=NWGVG25LCIVX type=write size=1589585 2023-12-03T11:30:48.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.036+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=1YH65ZQJGOTR type=del size=1556362 2023-12-03T11:30:48.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.037+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~W0IAO01A8EDP.0 size=1048576 2023-12-03T11:30:48.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.038+0000 7fc553ded780 1 < completed part key=_multipart_PL22GPUFN3O0.2~W0IAO01A8EDP.1 size=777830 2023-12-03T11:30:48.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.040+0000 7fc553ded780 1 < completed multipart upload key=PL22GPUFN3O0 upload=W0IAO01A8EDP tag=5IE9NUBJ04FC type=write size=2874982 2023-12-03T11:30:48.042 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.041+0000 7fc553ded780 1 > prepared operation key=588S2ASN9I0W tag=GVPHEK548E2Y type=del size=2005176 2023-12-03T11:30:48.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.043+0000 7fc553ded780 1 < suggested remove operation key=ZUNOFY5N3J3H tag=ZMNFN1HLOZ7B type=write 2023-12-03T11:30:48.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.046+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~G9SB7YJD8C59.0 size=1048576 2023-12-03T11:30:48.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.047+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~G9SB7YJD8C59.1 size=696389 2023-12-03T11:30:48.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.047+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=G9SB7YJD8C59 tag=6MPNR29847YU type=write size=2793541 2023-12-03T11:30:48.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.050+0000 7fc553ded780 1 < suggested update operation key=EK7V6U9DSLD0 tag=HSANIJVNRS01 type=del size=988238 2023-12-03T11:30:48.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.051+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=QDYJEPAN7MOH type=write size=674881 2023-12-03T11:30:48.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.052+0000 7fc553ded780 1 < canceled operation key=JUYA4LXEQ4US tag=0V6O1TQZHJFV type=del size=2931329 2023-12-03T11:30:48.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.053+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=588ZOFUFH7PI type=del size=3751586 2023-12-03T11:30:48.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.054+0000 7fc553ded780 1 < completed write operation key=SWTRHXTAR0A1 tag=HIK85GNUY3XI type=write size=859 2023-12-03T11:30:48.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.057+0000 7fc553ded780 1 > prepared part key=_multipart_FP2UZMDKYTD1.2~V4ZSC5KCBRK3.0 size=38934 2023-12-03T11:30:48.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.057+0000 7fc553ded780 1 > prepared multipart upload key=FP2UZMDKYTD1 upload=V4ZSC5KCBRK3 tag=3PAK4JBRGMY1 type=write size=1087510 2023-12-03T11:30:48.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.059+0000 7fc553ded780 1 < suggested remove operation key=EK7V6U9DSLD0 tag=S5YO0FNCTD9O type=del 2023-12-03T11:30:48.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.060+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=1JXXVXL2ZYD2 type=del size=2382495 2023-12-03T11:30:48.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.062+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.0 size=1048576 2023-12-03T11:30:48.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.063+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.1 size=1048576 2023-12-03T11:30:48.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.064+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~9DDXS3NZXR3I.2 size=494253 2023-12-03T11:30:48.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.066+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=9DDXS3NZXR3I tag=E8U4YOF0BFHH type=write size=3639981 2023-12-03T11:30:48.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.067+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=96W02SQDK19W type=del size=1766696 2023-12-03T11:30:48.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.068+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.0 size=1048576 2023-12-03T11:30:48.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.069+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.1 size=1048576 2023-12-03T11:30:48.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.070+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~22V94M6VB2DH.2 size=714927 2023-12-03T11:30:48.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.072+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=22V94M6VB2DH tag=YFYPEN5S6F9E type=write size=3860655 2023-12-03T11:30:48.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.074+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=8Q225UGSUEJE type=del size=568332 2023-12-03T11:30:48.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.075+0000 7fc553ded780 1 < completed delete operation key=ZHK2M78A0SLQ tag=IQHHUC3XG7KG type=del 2023-12-03T11:30:48.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.076+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=EZCDIPNVLXW2 type=del size=549918 2023-12-03T11:30:48.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.077+0000 7fc553ded780 1 < completed part key=_multipart_G1OBBD75M5ZQ.2~DZUWRZG10900.0 size=1048576 2023-12-03T11:30:48.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.077+0000 7fc553ded780 1 < canceled part key=_multipart_G1OBBD75M5ZQ.2~DZUWRZG10900.1 size=831436 2023-12-03T11:30:48.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.080+0000 7fc553ded780 1 < canceled multipart upload key=G1OBBD75M5ZQ upload=DZUWRZG10900 tag=Z1RV3K7V8SVO type=write size=2928588 2023-12-03T11:30:48.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.081+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=9S83OJZ6F3D0 type=del size=2810226 2023-12-03T11:30:48.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.082+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=UF6UN6YJFNJ0 type=del 2023-12-03T11:30:48.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.083+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=XDYVTXNCK49H type=del size=3491555 2023-12-03T11:30:48.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.085+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.0 size=1048576 2023-12-03T11:30:48.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.085+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.1 size=1048576 2023-12-03T11:30:48.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.087+0000 7fc553ded780 1 < completed part key=_multipart_JUYA4LXEQ4US.2~KSK3HJNTCKVO.2 size=649466 2023-12-03T11:30:48.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.089+0000 7fc553ded780 1 < completed multipart upload key=JUYA4LXEQ4US upload=KSK3HJNTCKVO tag=2VE4U5HRRY7Z type=write size=3795194 2023-12-03T11:30:48.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.090+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=CA7L3N8SCJ07 type=del size=3102684 2023-12-03T11:30:48.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.091+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.0 size=1048576 2023-12-03T11:30:48.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.092+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.1 size=1048576 2023-12-03T11:30:48.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.093+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~1U9QMDGTGKDF.2 size=941511 2023-12-03T11:30:48.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.095+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=1U9QMDGTGKDF tag=32MNNDFWFAPX type=write size=4087239 2023-12-03T11:30:48.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.097+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~FTKFAEY31NVC.0 size=1048576 2023-12-03T11:30:48.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.098+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~FTKFAEY31NVC.1 size=484171 2023-12-03T11:30:48.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.098+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=FTKFAEY31NVC tag=HV6KHYG4MZV0 type=write size=2581323 2023-12-03T11:30:48.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.100+0000 7fc553ded780 1 < completed part key=_multipart_X741JIRT7RIG.2~57EVTPQB33P8.0 size=948658 2023-12-03T11:30:48.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.102+0000 7fc553ded780 1 < completed multipart upload key=X741JIRT7RIG upload=57EVTPQB33P8 tag=TRCYN42AXVKZ type=write size=1997234 2023-12-03T11:30:48.104 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.103+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=CBSG88JS95OX type=del size=1422070 2023-12-03T11:30:48.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.104+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~0TEE365XUPP7.0 size=1048576 2023-12-03T11:30:48.106 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.105+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~0TEE365XUPP7.1 size=871903 2023-12-03T11:30:48.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.107+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=0TEE365XUPP7 tag=QKR2X2EP778K type=write size=2969055 2023-12-03T11:30:48.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.110+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.0 size=1048576 2023-12-03T11:30:48.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.111+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.1 size=1048576 2023-12-03T11:30:48.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.111+0000 7fc553ded780 1 > prepared part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.2 size=493988 2023-12-03T11:30:48.112 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.111+0000 7fc553ded780 1 > prepared multipart upload key=FC4WU6SMNEWV upload=EAZO0JFANJJ5 tag=0HQUQ9SK2W9C type=write size=3639716 2023-12-03T11:30:48.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.114+0000 7fc553ded780 1 < suggested remove operation key=588S2ASN9I0W tag=1YH65ZQJGOTR type=del 2023-12-03T11:30:48.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.115+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=0XFKS1GSZ4AI type=write size=629996 2023-12-03T11:30:48.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.117+0000 7fc553ded780 -1 < failed to complete operation key=588S2ASN9I0W tag=GVPHEK548E2Y type=del size=2005176: (22) Invalid argument 2023-12-03T11:30:48.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.119+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.0 size=1048576 2023-12-03T11:30:48.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.120+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.1 size=1048576 2023-12-03T11:30:48.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.121+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.2 size=584165 2023-12-03T11:30:48.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.121+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=7XJW45MT3HOI tag=A35DQ2HYAGE5 type=write size=3729893 2023-12-03T11:30:48.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.122+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~G9SB7YJD8C59.0 size=1048576 2023-12-03T11:30:48.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.123+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~G9SB7YJD8C59.1 size=696389 2023-12-03T11:30:48.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.126+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=G9SB7YJD8C59 tag=6MPNR29847YU type=write size=2793541 2023-12-03T11:30:48.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.127+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=LCMKP2WTHZ51 type=write size=209855 2023-12-03T11:30:48.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.129+0000 7fc553ded780 1 < suggested remove operation key=IRMADF9MXXBZ tag=QDYJEPAN7MOH type=write 2023-12-03T11:30:48.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.131+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=O4GGV0I96KYF type=del size=776609 2023-12-03T11:30:48.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.132+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=588ZOFUFH7PI type=del 2023-12-03T11:30:48.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.133+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=NJKJ4Z7KFH7F type=del size=1933077 2023-12-03T11:30:48.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.134+0000 7fc553ded780 1 < completed part key=_multipart_FP2UZMDKYTD1.2~V4ZSC5KCBRK3.0 size=38934 2023-12-03T11:30:48.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.136+0000 7fc553ded780 1 < completed multipart upload key=FP2UZMDKYTD1 upload=V4ZSC5KCBRK3 tag=3PAK4JBRGMY1 type=write size=1087510 2023-12-03T11:30:48.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.137+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=K7U4Z3DIRXZY type=write size=847578 2023-12-03T11:30:48.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.138+0000 7fc553ded780 1 < completed delete operation key=4TWM1QM1OHOF tag=1JXXVXL2ZYD2 type=del 2023-12-03T11:30:48.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.141+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=3F8JH5KK4O0D type=del size=1759261 2023-12-03T11:30:48.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.143+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=96W02SQDK19W type=del 2023-12-03T11:30:48.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.145+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=PDDIXRPRQOPJ type=del size=2186939 2023-12-03T11:30:48.148 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.147+0000 7fc553ded780 1 < suggested update operation key=PL22GPUFN3O0 tag=8Q225UGSUEJE type=del size=568332 2023-12-03T11:30:48.149 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.148+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=38HTFMRUN5N1 type=del size=2412720 2023-12-03T11:30:48.152 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.151+0000 7fc553ded780 1 < suggested update operation key=6OW429S51UWN tag=EZCDIPNVLXW2 type=del size=549918 2023-12-03T11:30:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.153+0000 7fc553ded780 1 > prepared part key=_multipart_C2TPF9Q5Z7XQ.2~MO133XPZMB7Y.0 size=438136 2023-12-03T11:30:48.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.153+0000 7fc553ded780 1 > prepared multipart upload key=C2TPF9Q5Z7XQ upload=MO133XPZMB7Y tag=KTPOHWQFRGDD type=write size=1486712 2023-12-03T11:30:48.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.154+0000 7fc553ded780 -1 < failed to complete operation key=PL22GPUFN3O0 tag=9S83OJZ6F3D0 type=del size=2810226: (22) Invalid argument 2023-12-03T11:30:48.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.156+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=NND0WJRGLB9V type=write size=695957 2023-12-03T11:30:48.158 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.157+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=XDYVTXNCK49H type=del 2023-12-03T11:30:48.160 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.160+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~K46CCOD82HT3.0 size=121265 2023-12-03T11:30:48.161 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.160+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=K46CCOD82HT3 tag=44NM24T4XVKT type=write size=1169841 2023-12-03T11:30:48.162 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.161+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=CA7L3N8SCJ07 type=del 2023-12-03T11:30:48.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.165+0000 7fc553ded780 1 > prepared part key=_multipart_I3U08QX8MLBD.2~XV4KOD3DPV10.0 size=481323 2023-12-03T11:30:48.166 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.165+0000 7fc553ded780 1 > prepared multipart upload key=I3U08QX8MLBD upload=XV4KOD3DPV10 tag=RQZDD2PIRJQJ type=write size=1529899 2023-12-03T11:30:48.168 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.167+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~FTKFAEY31NVC.0 size=1048576 2023-12-03T11:30:48.169 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.168+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~FTKFAEY31NVC.1 size=484171 2023-12-03T11:30:48.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.171+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=FTKFAEY31NVC tag=HV6KHYG4MZV0 type=write size=2581323 2023-12-03T11:30:48.173 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.172+0000 7fc553ded780 1 > prepared operation key=6OW429S51UWN tag=88P0U2G3VSLY type=del size=1028340 2023-12-03T11:30:48.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.174+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=CBSG88JS95OX type=del 2023-12-03T11:30:48.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.177+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~5WJRW0L21KA5.0 size=870274 2023-12-03T11:30:48.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.177+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=5WJRW0L21KA5 tag=FTRJ3ZJXZXJO type=write size=1918850 2023-12-03T11:30:48.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.177+0000 7fc553ded780 1 < canceled part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.0 size=1048576 2023-12-03T11:30:48.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.178+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.1 size=1048576 2023-12-03T11:30:48.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.179+0000 7fc553ded780 1 < completed part key=_multipart_FC4WU6SMNEWV.2~EAZO0JFANJJ5.2 size=493988 2023-12-03T11:30:48.182 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.181+0000 7fc553ded780 1 < canceled multipart upload key=FC4WU6SMNEWV upload=EAZO0JFANJJ5 tag=0HQUQ9SK2W9C type=write size=3639716 2023-12-03T11:30:48.183 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.182+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=9NM7U6ECKDWO type=del size=592908 2023-12-03T11:30:48.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.186+0000 7fc553ded780 1 < suggested remove operation key=JUYA4LXEQ4US tag=0XFKS1GSZ4AI type=write 2023-12-03T11:30:48.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.187+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=57TDPPVJ38B8 type=del size=538795 2023-12-03T11:30:48.189 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.188+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.0 size=1048576 2023-12-03T11:30:48.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.189+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.1 size=1048576 2023-12-03T11:30:48.191 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.190+0000 7fc553ded780 1 < completed part key=_multipart_WLTMDGRBF4C9.2~7XJW45MT3HOI.2 size=584165 2023-12-03T11:30:48.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.193+0000 7fc553ded780 1 < completed multipart upload key=WLTMDGRBF4C9 upload=7XJW45MT3HOI tag=A35DQ2HYAGE5 type=write size=3729893 2023-12-03T11:30:48.195 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.194+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=SSBKSPVBAC51 type=del size=1063424 2023-12-03T11:30:48.196 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.195+0000 7fc553ded780 1 < completed write operation key=RO0KGI2QMA71 tag=LCMKP2WTHZ51 type=write size=209855 2023-12-03T11:30:48.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.196+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=78OO084BYBCL type=del size=2522138 2023-12-03T11:30:48.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.198+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=O4GGV0I96KYF type=del 2023-12-03T11:30:48.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.200+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.0 size=1048576 2023-12-03T11:30:48.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.201+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.1 size=1048576 2023-12-03T11:30:48.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.203+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.2 size=421099 2023-12-03T11:30:48.204 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.203+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=HOCA2RX0OS2Y tag=BG5T14X9CHPV type=write size=3566827 2023-12-03T11:30:48.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.204+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=NJKJ4Z7KFH7F type=del 2023-12-03T11:30:48.206 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.205+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=WYIZHXKUAC33 type=write size=932006 2023-12-03T11:30:48.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.206+0000 7fc553ded780 1 < completed write operation key=L4ME9RYRR4NG tag=K7U4Z3DIRXZY type=write size=847578 2023-12-03T11:30:48.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.209+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.0 size=1048576 2023-12-03T11:30:48.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.210+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.1 size=1048576 2023-12-03T11:30:48.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.211+0000 7fc553ded780 1 > prepared part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.2 size=900266 2023-12-03T11:30:48.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.211+0000 7fc553ded780 1 > prepared multipart upload key=0H5OZMBMLQFJ upload=Z2HTHGSK2M6W tag=Y06T72UGUQ9I type=write size=4045994 2023-12-03T11:30:48.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.212+0000 7fc553ded780 1 < canceled operation key=JUDXD9DL4N43 tag=3F8JH5KK4O0D type=del size=1759261 2023-12-03T11:30:48.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.214+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=GMDV2PZX3GX0 type=del size=258323 2023-12-03T11:30:48.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.216+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=PDDIXRPRQOPJ type=del size=2186939 2023-12-03T11:30:48.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.218+0000 7fc553ded780 1 > prepared part key=_multipart_ZHK2M78A0SLQ.2~JV4XFEIGB6C9.0 size=226743 2023-12-03T11:30:48.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.218+0000 7fc553ded780 1 > prepared multipart upload key=ZHK2M78A0SLQ upload=JV4XFEIGB6C9 tag=T470WNK6Y2ZW type=write size=1275319 2023-12-03T11:30:48.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.219+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=38HTFMRUN5N1 type=del 2023-12-03T11:30:48.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.222+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.0 size=1048576 2023-12-03T11:30:48.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.223+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.1 size=1048576 2023-12-03T11:30:48.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.224+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.2 size=547201 2023-12-03T11:30:48.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.224+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=8256YMTH5Q6O tag=HJUY0I2GI5P4 type=write size=3692929 2023-12-03T11:30:48.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.224+0000 7fc553ded780 1 < canceled part key=_multipart_C2TPF9Q5Z7XQ.2~MO133XPZMB7Y.0 size=438136 2023-12-03T11:30:48.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.226+0000 7fc553ded780 1 < canceled multipart upload key=C2TPF9Q5Z7XQ upload=MO133XPZMB7Y tag=KTPOHWQFRGDD type=write size=1486712 2023-12-03T11:30:48.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.228+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~UCE4F3SZM7MV.0 size=1048576 2023-12-03T11:30:48.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.229+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~UCE4F3SZM7MV.1 size=567841 2023-12-03T11:30:48.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.229+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=UCE4F3SZM7MV tag=0N0OQ86YH1Y7 type=write size=2664993 2023-12-03T11:30:48.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.231+0000 7fc553ded780 1 < canceled operation key=PL22GPUFN3O0 tag=NND0WJRGLB9V type=write size=695957 2023-12-03T11:30:48.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.232+0000 7fc553ded780 1 > prepared operation key=J664UUE697Q8 tag=8H8HFU34SNA0 type=del size=3859522 2023-12-03T11:30:48.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.234+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~K46CCOD82HT3.0 size=121265 2023-12-03T11:30:48.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.236+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=K46CCOD82HT3 tag=44NM24T4XVKT type=write size=1169841 2023-12-03T11:30:48.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.237+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=B62HY49ITINU type=del size=191017 2023-12-03T11:30:48.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.238+0000 7fc553ded780 1 < completed part key=_multipart_I3U08QX8MLBD.2~XV4KOD3DPV10.0 size=481323 2023-12-03T11:30:48.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.240+0000 7fc553ded780 1 < completed multipart upload key=I3U08QX8MLBD upload=XV4KOD3DPV10 tag=RQZDD2PIRJQJ type=write size=1529899 2023-12-03T11:30:48.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.241+0000 7fc553ded780 1 > prepared operation key=EK7V6U9DSLD0 tag=LA8S06PX5LRI type=del size=4075385 2023-12-03T11:30:48.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.242+0000 7fc553ded780 1 < completed delete operation key=6OW429S51UWN tag=88P0U2G3VSLY type=del 2023-12-03T11:30:48.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.244+0000 7fc553ded780 1 > prepared operation key=UKAL8FCVIRSH tag=KTMNO0LSJLHW type=del size=957732 2023-12-03T11:30:48.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.245+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~5WJRW0L21KA5.0 size=870274 2023-12-03T11:30:48.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.247+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=5WJRW0L21KA5 tag=FTRJ3ZJXZXJO type=write size=1918850 2023-12-03T11:30:48.250 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.249+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=I5R81FK6PO1D type=del size=2061657 2023-12-03T11:30:48.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.251+0000 7fc553ded780 1 < suggested update operation key=FC4WU6SMNEWV tag=9NM7U6ECKDWO type=del size=592908 2023-12-03T11:30:48.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.253+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=71YN74X5ZQOC type=del size=2231008 2023-12-03T11:30:48.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.254+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=57TDPPVJ38B8 type=del 2023-12-03T11:30:48.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.255+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=HZ2X5S4UDR2O type=del size=1709457 2023-12-03T11:30:48.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.257+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=SSBKSPVBAC51 type=del 2023-12-03T11:30:48.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.259+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~9MOHIRHH9CKQ.0 size=1048576 2023-12-03T11:30:48.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.260+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~9MOHIRHH9CKQ.1 size=596920 2023-12-03T11:30:48.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.260+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=9MOHIRHH9CKQ tag=1H64J3N9MEV4 type=write size=2694072 2023-12-03T11:30:48.262 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.261+0000 7fc553ded780 1 < completed delete operation key=8GRA2MUBGWJI tag=78OO084BYBCL type=del 2023-12-03T11:30:48.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.262+0000 7fc553ded780 1 > prepared operation key=WLTMDGRBF4C9 tag=1FO4GRQWQUNV type=write size=964771 2023-12-03T11:30:48.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.263+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.0 size=1048576 2023-12-03T11:30:48.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.264+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.1 size=1048576 2023-12-03T11:30:48.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.265+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~HOCA2RX0OS2Y.2 size=421099 2023-12-03T11:30:48.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.268+0000 7fc553ded780 1 < completed multipart upload key=4TWM1QM1OHOF upload=HOCA2RX0OS2Y tag=BG5T14X9CHPV type=write size=3566827 2023-12-03T11:30:48.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.269+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=3S2V5R9HSJO3 type=del size=3607676 2023-12-03T11:30:48.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.270+0000 7fc553ded780 -1 < failed to complete operation key=FC4WU6SMNEWV tag=WYIZHXKUAC33 type=write size=932006: (22) Invalid argument 2023-12-03T11:30:48.272 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.271+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=WGRC8B4FMEBR type=del size=1865457 2023-12-03T11:30:48.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.273+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.0 size=1048576 2023-12-03T11:30:48.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.274+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.1 size=1048576 2023-12-03T11:30:48.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.275+0000 7fc553ded780 1 < completed part key=_multipart_0H5OZMBMLQFJ.2~Z2HTHGSK2M6W.2 size=900266 2023-12-03T11:30:48.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.277+0000 7fc553ded780 1 < completed multipart upload key=0H5OZMBMLQFJ upload=Z2HTHGSK2M6W tag=Y06T72UGUQ9I type=write size=4045994 2023-12-03T11:30:48.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.278+0000 7fc553ded780 1 > prepared operation key=IRMADF9MXXBZ tag=44P0ZS90KX44 type=del size=327379 2023-12-03T11:30:48.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.281+0000 7fc553ded780 1 < suggested remove operation key=ZUNOFY5N3J3H tag=GMDV2PZX3GX0 type=del 2023-12-03T11:30:48.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.283+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.0 size=1048576 2023-12-03T11:30:48.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.284+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.1 size=1048576 2023-12-03T11:30:48.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.285+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.2 size=645539 2023-12-03T11:30:48.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.285+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=JRFF2VUYSLX7 tag=9AEJLP6THOWP type=write size=3791267 2023-12-03T11:30:48.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.287+0000 7fc553ded780 1 < completed part key=_multipart_ZHK2M78A0SLQ.2~JV4XFEIGB6C9.0 size=226743 2023-12-03T11:30:48.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.289+0000 7fc553ded780 1 < completed multipart upload key=ZHK2M78A0SLQ upload=JV4XFEIGB6C9 tag=T470WNK6Y2ZW type=write size=1275319 2023-12-03T11:30:48.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.290+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=GT0KAUZRZMMW type=write size=882581 2023-12-03T11:30:48.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.292+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.0 size=1048576 2023-12-03T11:30:48.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.293+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.1 size=1048576 2023-12-03T11:30:48.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.294+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~8256YMTH5Q6O.2 size=547201 2023-12-03T11:30:48.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.296+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=8256YMTH5Q6O tag=HJUY0I2GI5P4 type=write size=3692929 2023-12-03T11:30:48.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.298+0000 7fc553ded780 1 > prepared part key=_multipart_WDV4CW863V33.2~N2O9LP5CC0KE.0 size=993530 2023-12-03T11:30:48.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.298+0000 7fc553ded780 1 > prepared multipart upload key=WDV4CW863V33 upload=N2O9LP5CC0KE tag=MWMESPXF70VS type=write size=2042106 2023-12-03T11:30:48.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.299+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~UCE4F3SZM7MV.0 size=1048576 2023-12-03T11:30:48.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.300+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~UCE4F3SZM7MV.1 size=567841 2023-12-03T11:30:48.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.302+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=UCE4F3SZM7MV tag=0N0OQ86YH1Y7 type=write size=2664993 2023-12-03T11:30:48.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.303+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=2QL6AW7PPR53 type=del size=523580 2023-12-03T11:30:48.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.304+0000 7fc553ded780 1 < completed delete operation key=J664UUE697Q8 tag=8H8HFU34SNA0 type=del 2023-12-03T11:30:48.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.306+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=15T23096INP6 type=del size=871096 2023-12-03T11:30:48.308 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.307+0000 7fc553ded780 -1 < failed to complete operation key=ZUNOFY5N3J3H tag=B62HY49ITINU type=del size=191017: (22) Invalid argument 2023-12-03T11:30:48.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.310+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.0 size=1048576 2023-12-03T11:30:48.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.311+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.1 size=1048576 2023-12-03T11:30:48.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.312+0000 7fc553ded780 1 > prepared part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.2 size=66819 2023-12-03T11:30:48.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.312+0000 7fc553ded780 1 > prepared multipart upload key=L4ME9RYRR4NG upload=8R1UI8C048RQ tag=324B89WY0WPN type=write size=3212547 2023-12-03T11:30:48.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.314+0000 7fc553ded780 1 < completed delete operation key=EK7V6U9DSLD0 tag=LA8S06PX5LRI type=del 2023-12-03T11:30:48.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.315+0000 7fc553ded780 1 > prepared operation key=ZHK2M78A0SLQ tag=PUIIELI9LQ3Z type=del size=3112928 2023-12-03T11:30:48.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.317+0000 7fc553ded780 1 < completed delete operation key=UKAL8FCVIRSH tag=KTMNO0LSJLHW type=del 2023-12-03T11:30:48.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.318+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=T4D3NTM705Y3 type=write size=753058 2023-12-03T11:30:48.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.320+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=I5R81FK6PO1D type=del 2023-12-03T11:30:48.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.321+0000 7fc553ded780 1 > prepared operation key=L4ME9RYRR4NG tag=CWZ9QVOOAVBO type=del size=2225570 2023-12-03T11:30:48.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.323+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=71YN74X5ZQOC type=del 2023-12-03T11:30:48.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.324+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=JD1T4WLMT5HC type=write size=503440 2023-12-03T11:30:48.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.326+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=HZ2X5S4UDR2O type=del 2023-12-03T11:30:48.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.328+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.0 size=1048576 2023-12-03T11:30:48.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.329+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.1 size=1048576 2023-12-03T11:30:48.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.330+0000 7fc553ded780 1 > prepared part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.2 size=921291 2023-12-03T11:30:48.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.330+0000 7fc553ded780 1 > prepared multipart upload key=4TWM1QM1OHOF upload=38L2BY46EGFZ tag=2GGN8X71H065 type=write size=4067019 2023-12-03T11:30:48.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.331+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~9MOHIRHH9CKQ.0 size=1048576 2023-12-03T11:30:48.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.332+0000 7fc553ded780 1 < completed part key=_multipart_IRMADF9MXXBZ.2~9MOHIRHH9CKQ.1 size=596920 2023-12-03T11:30:48.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.334+0000 7fc553ded780 1 < completed multipart upload key=IRMADF9MXXBZ upload=9MOHIRHH9CKQ tag=1H64J3N9MEV4 type=write size=2694072 2023-12-03T11:30:48.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.336+0000 7fc553ded780 1 > prepared part key=_multipart_588S2ASN9I0W.2~GA9LFC5L7MOT.0 size=208303 2023-12-03T11:30:48.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.336+0000 7fc553ded780 1 > prepared multipart upload key=588S2ASN9I0W upload=GA9LFC5L7MOT tag=ZL66KVY4Z95B type=write size=1256879 2023-12-03T11:30:48.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.337+0000 7fc553ded780 1 < completed write operation key=WLTMDGRBF4C9 tag=1FO4GRQWQUNV type=write size=964771 2023-12-03T11:30:48.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.339+0000 7fc553ded780 1 > prepared operation key=PL22GPUFN3O0 tag=YZ66XRSHREL2 type=del size=3392600 2023-12-03T11:30:48.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.340+0000 7fc553ded780 1 < completed delete operation key=CZP8OQRU641T tag=3S2V5R9HSJO3 type=del 2023-12-03T11:30:48.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.341+0000 7fc553ded780 1 > prepared operation key=FP2UZMDKYTD1 tag=M0CW95G2X0CI type=del size=383034 2023-12-03T11:30:48.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.343+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=WGRC8B4FMEBR type=del 2023-12-03T11:30:48.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.345+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.0 size=1048576 2023-12-03T11:30:48.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.346+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.1 size=1048576 2023-12-03T11:30:48.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.347+0000 7fc553ded780 1 > prepared part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.2 size=637689 2023-12-03T11:30:48.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.347+0000 7fc553ded780 1 > prepared multipart upload key=RO0KGI2QMA71 upload=9GP9CRDI7GBN tag=XN3JO41QGPUL type=write size=3783417 2023-12-03T11:30:48.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.348+0000 7fc553ded780 1 < completed delete operation key=IRMADF9MXXBZ tag=44P0ZS90KX44 type=del 2023-12-03T11:30:48.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.351+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I6U4QRL70KMH.0 size=1048576 2023-12-03T11:30:48.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.352+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~I6U4QRL70KMH.1 size=124002 2023-12-03T11:30:48.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.352+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=I6U4QRL70KMH tag=93POT72KU6H6 type=write size=2221154 2023-12-03T11:30:48.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.352+0000 7fc553ded780 1 < canceled part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.0 size=1048576 2023-12-03T11:30:48.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.353+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.1 size=1048576 2023-12-03T11:30:48.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.355+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~JRFF2VUYSLX7.2 size=645539 2023-12-03T11:30:48.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.357+0000 7fc553ded780 1 < canceled multipart upload key=L4ME9RYRR4NG upload=JRFF2VUYSLX7 tag=9AEJLP6THOWP type=write size=3791267 2023-12-03T11:30:48.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.358+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=3GJ49934VXZ3 type=del size=2981811 2023-12-03T11:30:48.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.359+0000 7fc553ded780 1 < canceled operation key=X741JIRT7RIG tag=GT0KAUZRZMMW type=write size=882581 2023-12-03T11:30:48.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.360+0000 7fc553ded780 1 > prepared operation key=RO0KGI2QMA71 tag=OH163H2GAA6V type=del size=3695409 2023-12-03T11:30:48.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.362+0000 7fc553ded780 1 < completed part key=_multipart_WDV4CW863V33.2~N2O9LP5CC0KE.0 size=993530 2023-12-03T11:30:48.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.364+0000 7fc553ded780 1 < completed multipart upload key=WDV4CW863V33 upload=N2O9LP5CC0KE tag=MWMESPXF70VS type=write size=2042106 2023-12-03T11:30:48.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.365+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=JTLLCLUJJAM1 type=del size=1578052 2023-12-03T11:30:48.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.366+0000 7fc553ded780 1 < canceled operation key=706CET5PSDEG tag=2QL6AW7PPR53 type=del size=523580 2023-12-03T11:30:48.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.367+0000 7fc553ded780 1 > prepared operation key=SWTRHXTAR0A1 tag=HJGI578HZGCT type=del size=1092394 2023-12-03T11:30:48.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.368+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=15T23096INP6 type=del 2023-12-03T11:30:48.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.370+0000 7fc553ded780 1 > prepared operation key=YE73HBF64CS7 tag=DF9JXJPHJ7CP type=write size=983281 2023-12-03T11:30:48.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.371+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.0 size=1048576 2023-12-03T11:30:48.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.372+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.1 size=1048576 2023-12-03T11:30:48.374 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.373+0000 7fc553ded780 1 < completed part key=_multipart_L4ME9RYRR4NG.2~8R1UI8C048RQ.2 size=66819 2023-12-03T11:30:48.376 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.375+0000 7fc553ded780 1 < completed multipart upload key=L4ME9RYRR4NG upload=8R1UI8C048RQ tag=324B89WY0WPN type=write size=3212547 2023-12-03T11:30:48.378 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.377+0000 7fc553ded780 1 > prepared operation key=JUDXD9DL4N43 tag=4D2PE4S8WXXC type=del size=2038879 2023-12-03T11:30:48.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.378+0000 7fc553ded780 1 < canceled operation key=ZHK2M78A0SLQ tag=PUIIELI9LQ3Z type=del size=3112928 2023-12-03T11:30:48.381 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.380+0000 7fc553ded780 1 > prepared operation key=0H5OZMBMLQFJ tag=D7HYGI4S3UVG type=del size=3809096 2023-12-03T11:30:48.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.381+0000 7fc553ded780 1 < completed write operation key=0H5OZMBMLQFJ tag=T4D3NTM705Y3 type=write size=753058 2023-12-03T11:30:48.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.382+0000 7fc553ded780 1 > prepared operation key=4TWM1QM1OHOF tag=O2QLZWD1O04T type=write size=144656 2023-12-03T11:30:48.385 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.384+0000 7fc553ded780 1 < completed delete operation key=L4ME9RYRR4NG tag=CWZ9QVOOAVBO type=del 2023-12-03T11:30:48.386 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.385+0000 7fc553ded780 1 > prepared operation key=C2TPF9Q5Z7XQ tag=SFTP91PZTGWZ type=del size=3740923 2023-12-03T11:30:48.387 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.386+0000 7fc553ded780 1 < completed write operation key=JUDXD9DL4N43 tag=JD1T4WLMT5HC type=write size=503440 2023-12-03T11:30:48.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.391+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~TC5FF19JM2SP.0 size=1048576 2023-12-03T11:30:48.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.393+0000 7fc553ded780 1 > prepared part key=_multipart_YE73HBF64CS7.2~TC5FF19JM2SP.1 size=744258 2023-12-03T11:30:48.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.393+0000 7fc553ded780 1 > prepared multipart upload key=YE73HBF64CS7 upload=TC5FF19JM2SP tag=SIZ8ZLZ2WSTX type=write size=2841410 2023-12-03T11:30:48.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.395+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.0 size=1048576 2023-12-03T11:30:48.396 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.395+0000 7fc553ded780 1 < canceled part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.1 size=1048576 2023-12-03T11:30:48.397 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.396+0000 7fc553ded780 1 < completed part key=_multipart_4TWM1QM1OHOF.2~38L2BY46EGFZ.2 size=921291 2023-12-03T11:30:48.399 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.398+0000 7fc553ded780 1 < canceled multipart upload key=4TWM1QM1OHOF upload=38L2BY46EGFZ tag=2GGN8X71H065 type=write size=4067019 2023-12-03T11:30:48.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.399+0000 7fc553ded780 1 > prepared operation key=X741JIRT7RIG tag=TFJC1NLW78MS type=del size=1680557 2023-12-03T11:30:48.400 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.399+0000 7fc553ded780 1 < canceled part key=_multipart_588S2ASN9I0W.2~GA9LFC5L7MOT.0 size=208303 2023-12-03T11:30:48.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.401+0000 7fc553ded780 1 < canceled multipart upload key=588S2ASN9I0W upload=GA9LFC5L7MOT tag=ZL66KVY4Z95B type=write size=1256879 2023-12-03T11:30:48.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.403+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=60FIT6YFL2MY type=write size=525050 2023-12-03T11:30:48.405 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.404+0000 7fc553ded780 1 < completed delete operation key=PL22GPUFN3O0 tag=YZ66XRSHREL2 type=del 2023-12-03T11:30:48.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.406+0000 7fc553ded780 1 > prepared operation key=I3U08QX8MLBD tag=YTV1560VD1T9 type=del size=2672903 2023-12-03T11:30:48.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.407+0000 7fc553ded780 1 < completed delete operation key=FP2UZMDKYTD1 tag=M0CW95G2X0CI type=del 2023-12-03T11:30:48.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.410+0000 7fc553ded780 1 > prepared part key=_multipart_TQRZT37AB9WC.2~CRDR23NY1KO0.0 size=879688 2023-12-03T11:30:48.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.410+0000 7fc553ded780 1 > prepared multipart upload key=TQRZT37AB9WC upload=CRDR23NY1KO0 tag=FDFBXTSYB050 type=write size=1928264 2023-12-03T11:30:48.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.411+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.0 size=1048576 2023-12-03T11:30:48.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.412+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.1 size=1048576 2023-12-03T11:30:48.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.413+0000 7fc553ded780 1 < completed part key=_multipart_RO0KGI2QMA71.2~9GP9CRDI7GBN.2 size=637689 2023-12-03T11:30:48.417 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.416+0000 7fc553ded780 1 < completed multipart upload key=RO0KGI2QMA71 upload=9GP9CRDI7GBN tag=XN3JO41QGPUL type=write size=3783417 2023-12-03T11:30:48.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.417+0000 7fc553ded780 1 > prepared operation key=706CET5PSDEG tag=5L6Z6HVTXI9K type=del size=3426919 2023-12-03T11:30:48.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.418+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I6U4QRL70KMH.0 size=1048576 2023-12-03T11:30:48.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.419+0000 7fc553ded780 1 < completed part key=_multipart_EK7V6U9DSLD0.2~I6U4QRL70KMH.1 size=124002 2023-12-03T11:30:48.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.422+0000 7fc553ded780 1 < completed multipart upload key=EK7V6U9DSLD0 upload=I6U4QRL70KMH tag=93POT72KU6H6 type=write size=2221154 2023-12-03T11:30:48.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.424+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~37AAW9G1TXT6.0 size=1048576 2023-12-03T11:30:48.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.425+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~37AAW9G1TXT6.1 size=379600 2023-12-03T11:30:48.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.425+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=37AAW9G1TXT6 tag=BTK37VZL30CF type=write size=2476752 2023-12-03T11:30:48.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.428+0000 7fc553ded780 1 < suggested remove operation key=TQRZT37AB9WC tag=3GJ49934VXZ3 type=del 2023-12-03T11:30:48.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.431+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.0 size=1048576 2023-12-03T11:30:48.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.432+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.1 size=1048576 2023-12-03T11:30:48.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.433+0000 7fc553ded780 1 > prepared part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.2 size=817584 2023-12-03T11:30:48.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.433+0000 7fc553ded780 1 > prepared multipart upload key=WOKG1DISIKK0 upload=JXVB9QFDRVOJ tag=LTYO9ZQVNH9B type=write size=3963312 2023-12-03T11:30:48.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.434+0000 7fc553ded780 1 < completed delete operation key=RO0KGI2QMA71 tag=OH163H2GAA6V type=del 2023-12-03T11:30:48.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.438+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~2ZR5HSNR0MUL.0 size=1048576 2023-12-03T11:30:48.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.439+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~2ZR5HSNR0MUL.1 size=687588 2023-12-03T11:30:48.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.439+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=2ZR5HSNR0MUL tag=2VQN8ZYBZXJB type=write size=2784740 2023-12-03T11:30:48.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.441+0000 7fc553ded780 1 < completed delete operation key=ZUNOFY5N3J3H tag=JTLLCLUJJAM1 type=del 2023-12-03T11:30:48.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.442+0000 7fc553ded780 1 > prepared operation key=JUYA4LXEQ4US tag=7GXD4MSUN49L type=del size=2451834 2023-12-03T11:30:48.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.444+0000 7fc553ded780 1 < completed delete operation key=SWTRHXTAR0A1 tag=HJGI578HZGCT type=del 2023-12-03T11:30:48.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.445+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=7TMP0MAMF8M2 type=del size=1894014 2023-12-03T11:30:48.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.447+0000 7fc553ded780 1 < completed write operation key=YE73HBF64CS7 tag=DF9JXJPHJ7CP type=write size=983281 2023-12-03T11:30:48.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.449+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~OJA558S9GFJL.0 size=1048576 2023-12-03T11:30:48.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.450+0000 7fc553ded780 1 > prepared part key=_multipart_6OW429S51UWN.2~OJA558S9GFJL.1 size=490064 2023-12-03T11:30:48.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.450+0000 7fc553ded780 1 > prepared multipart upload key=6OW429S51UWN upload=OJA558S9GFJL tag=DYEE7BUL6N0P type=write size=2587216 2023-12-03T11:30:48.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.451+0000 7fc553ded780 1 < completed delete operation key=JUDXD9DL4N43 tag=4D2PE4S8WXXC type=del 2023-12-03T11:30:48.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.453+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=8A4U0OD0XA7K type=del size=4043264 2023-12-03T11:30:48.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.455+0000 7fc553ded780 1 < suggested update operation key=0H5OZMBMLQFJ tag=D7HYGI4S3UVG type=del size=3809096 2023-12-03T11:30:48.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.457+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=YB4FIQTZDZ0N type=del size=3887136 2023-12-03T11:30:48.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.458+0000 7fc553ded780 1 < completed write operation key=4TWM1QM1OHOF tag=O2QLZWD1O04T type=write size=144656 2023-12-03T11:30:48.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.461+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.0 size=1048576 2023-12-03T11:30:48.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.461+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.1 size=1048576 2023-12-03T11:30:48.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.462+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.2 size=439612 2023-12-03T11:30:48.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.462+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=NE4HE1E0CDOC tag=XCWCLK7MY8HM type=write size=3585340 2023-12-03T11:30:48.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.464+0000 7fc553ded780 1 < completed delete operation key=C2TPF9Q5Z7XQ tag=SFTP91PZTGWZ type=del 2023-12-03T11:30:48.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.465+0000 7fc553ded780 1 > prepared operation key=WDV4CW863V33 tag=UC51SO6VGA1O type=del size=4001482 2023-12-03T11:30:48.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.465+0000 7fc553ded780 1 < canceled part key=_multipart_YE73HBF64CS7.2~TC5FF19JM2SP.0 size=1048576 2023-12-03T11:30:48.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.466+0000 7fc553ded780 1 < completed part key=_multipart_YE73HBF64CS7.2~TC5FF19JM2SP.1 size=744258 2023-12-03T11:30:48.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.468+0000 7fc553ded780 1 < canceled multipart upload key=YE73HBF64CS7 upload=TC5FF19JM2SP tag=SIZ8ZLZ2WSTX type=write size=2841410 2023-12-03T11:30:48.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.470+0000 7fc553ded780 1 > prepared part key=_multipart_706CET5PSDEG.2~M2ZLZSSO2UT9.0 size=292631 2023-12-03T11:30:48.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.470+0000 7fc553ded780 1 > prepared multipart upload key=706CET5PSDEG upload=M2ZLZSSO2UT9 tag=TNJ6O7VT3UTJ type=write size=1341207 2023-12-03T11:30:48.472 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.471+0000 7fc553ded780 1 < completed delete operation key=X741JIRT7RIG tag=TFJC1NLW78MS type=del 2023-12-03T11:30:48.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.473+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~48IEP2XBD9ZN.0 size=1048576 2023-12-03T11:30:48.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.474+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~48IEP2XBD9ZN.1 size=1048576 2023-12-03T11:30:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.475+0000 7fc553ded780 1 > prepared part key=_multipart_WLTMDGRBF4C9.2~48IEP2XBD9ZN.2 size=948334 2023-12-03T11:30:48.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.475+0000 7fc553ded780 1 > prepared multipart upload key=WLTMDGRBF4C9 upload=48IEP2XBD9ZN tag=MU1EY071XCSG type=write size=4094062 2023-12-03T11:30:48.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.477+0000 7fc553ded780 1 < suggested remove operation key=FC4WU6SMNEWV tag=60FIT6YFL2MY type=write 2023-12-03T11:30:48.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.478+0000 7fc553ded780 1 > prepared operation key=CZP8OQRU641T tag=2SZEYMMG2SWX type=del size=175753 2023-12-03T11:30:48.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.480+0000 7fc553ded780 1 < completed delete operation key=I3U08QX8MLBD tag=YTV1560VD1T9 type=del 2023-12-03T11:30:48.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.482+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~843O18MGKDI1.0 size=1048576 2023-12-03T11:30:48.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.482+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~843O18MGKDI1.1 size=1048576 2023-12-03T11:30:48.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.483+0000 7fc553ded780 1 > prepared part key=_multipart_EK7V6U9DSLD0.2~843O18MGKDI1.2 size=556083 2023-12-03T11:30:48.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.483+0000 7fc553ded780 1 > prepared multipart upload key=EK7V6U9DSLD0 upload=843O18MGKDI1 tag=SYFP085N242Y type=write size=3701811 2023-12-03T11:30:48.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.485+0000 7fc553ded780 1 < completed part key=_multipart_TQRZT37AB9WC.2~CRDR23NY1KO0.0 size=879688 2023-12-03T11:30:48.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.487+0000 7fc553ded780 1 < completed multipart upload key=TQRZT37AB9WC upload=CRDR23NY1KO0 tag=FDFBXTSYB050 type=write size=1928264 2023-12-03T11:30:48.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.489+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~R3ZB2XCAMZJK.0 size=1048576 2023-12-03T11:30:48.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.490+0000 7fc553ded780 1 > prepared part key=_multipart_IRMADF9MXXBZ.2~R3ZB2XCAMZJK.1 size=728244 2023-12-03T11:30:48.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.490+0000 7fc553ded780 1 > prepared multipart upload key=IRMADF9MXXBZ upload=R3ZB2XCAMZJK tag=ENVO72CN8X0W type=write size=2825396 2023-12-03T11:30:48.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.491+0000 7fc553ded780 1 < completed delete operation key=706CET5PSDEG tag=5L6Z6HVTXI9K type=del 2023-12-03T11:30:48.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.493+0000 7fc553ded780 1 > prepared operation key=FC4WU6SMNEWV tag=2VA0EB7YXNIJ type=del size=946653 2023-12-03T11:30:48.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.495+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~37AAW9G1TXT6.0 size=1048576 2023-12-03T11:30:48.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.496+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~37AAW9G1TXT6.1 size=379600 2023-12-03T11:30:48.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.498+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=37AAW9G1TXT6 tag=BTK37VZL30CF type=write size=2476752 2023-12-03T11:30:48.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.501+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~4QZ8HDL1PY9T.0 size=1048576 2023-12-03T11:30:48.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.502+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~4QZ8HDL1PY9T.1 size=1048576 2023-12-03T11:30:48.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.503+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~4QZ8HDL1PY9T.2 size=550409 2023-12-03T11:30:48.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.503+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=4QZ8HDL1PY9T tag=Z1VV3PKO5ZTZ type=write size=3696137 2023-12-03T11:30:48.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.504+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.0 size=1048576 2023-12-03T11:30:48.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.504+0000 7fc553ded780 1 < canceled part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.1 size=1048576 2023-12-03T11:30:48.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.505+0000 7fc553ded780 1 < completed part key=_multipart_WOKG1DISIKK0.2~JXVB9QFDRVOJ.2 size=817584 2023-12-03T11:30:48.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.507+0000 7fc553ded780 1 < canceled multipart upload key=WOKG1DISIKK0 upload=JXVB9QFDRVOJ tag=LTYO9ZQVNH9B type=write size=3963312 2023-12-03T11:30:48.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.509+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OXU6P15CHOXO.0 size=1048576 2023-12-03T11:30:48.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.510+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OXU6P15CHOXO.1 size=1048576 2023-12-03T11:30:48.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.513+0000 7fc553ded780 1 > prepared part key=_multipart_LRN5NYUZ5JT9.2~OXU6P15CHOXO.2 size=648169 2023-12-03T11:30:48.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.513+0000 7fc553ded780 1 > prepared multipart upload key=LRN5NYUZ5JT9 upload=OXU6P15CHOXO tag=IYMUFSD1BHE1 type=write size=3793897 2023-12-03T11:30:48.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.515+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~2ZR5HSNR0MUL.0 size=1048576 2023-12-03T11:30:48.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.516+0000 7fc553ded780 1 < completed part key=_multipart_LRN5NYUZ5JT9.2~2ZR5HSNR0MUL.1 size=687588 2023-12-03T11:30:48.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.518+0000 7fc553ded780 1 < completed multipart upload key=LRN5NYUZ5JT9 upload=2ZR5HSNR0MUL tag=2VQN8ZYBZXJB type=write size=2784740 2023-12-03T11:30:48.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.520+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~ON92CJXKFIR1.0 size=1048576 2023-12-03T11:30:48.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.521+0000 7fc553ded780 1 > prepared part key=_multipart_8GRA2MUBGWJI.2~ON92CJXKFIR1.1 size=901938 2023-12-03T11:30:48.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.521+0000 7fc553ded780 1 > prepared multipart upload key=8GRA2MUBGWJI upload=ON92CJXKFIR1 tag=51AV0VLBT4BV type=write size=2999090 2023-12-03T11:30:48.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.523+0000 7fc553ded780 1 < suggested update operation key=JUYA4LXEQ4US tag=7GXD4MSUN49L type=del size=2451834 2023-12-03T11:30:48.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.525+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=U7M2LSTZLFKH type=del size=607357 2023-12-03T11:30:48.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.526+0000 7fc553ded780 1 < completed delete operation key=TQRZT37AB9WC tag=7TMP0MAMF8M2 type=del 2023-12-03T11:30:48.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.527+0000 7fc553ded780 1 > prepared operation key=8GRA2MUBGWJI tag=X7Q3WPMLH312 type=del size=1397675 2023-12-03T11:30:48.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.528+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~OJA558S9GFJL.0 size=1048576 2023-12-03T11:30:48.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.529+0000 7fc553ded780 1 < completed part key=_multipart_6OW429S51UWN.2~OJA558S9GFJL.1 size=490064 2023-12-03T11:30:48.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.531+0000 7fc553ded780 1 < completed multipart upload key=6OW429S51UWN upload=OJA558S9GFJL tag=DYEE7BUL6N0P type=write size=2587216 2023-12-03T11:30:48.534 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.532+0000 7fc553ded780 1 > prepared operation key=ZUNOFY5N3J3H tag=2JHC8AR0DUJ8 type=del size=1796397 2023-12-03T11:30:48.536 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.535+0000 7fc553ded780 1 < suggested remove operation key=CZP8OQRU641T tag=8A4U0OD0XA7K type=del 2023-12-03T11:30:48.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.537+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~V0RZR0TJ0L13.0 size=1048576 2023-12-03T11:30:48.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.538+0000 7fc553ded780 1 > prepared part key=_multipart_JUYA4LXEQ4US.2~V0RZR0TJ0L13.1 size=452560 2023-12-03T11:30:48.539 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.538+0000 7fc553ded780 1 > prepared multipart upload key=JUYA4LXEQ4US upload=V0RZR0TJ0L13 tag=I5DWZZO242JT type=write size=2549712 2023-12-03T11:30:48.540 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.539+0000 7fc553ded780 1 < completed delete operation key=G1OBBD75M5ZQ tag=YB4FIQTZDZ0N type=del 2023-12-03T11:30:48.541 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.540+0000 7fc553ded780 1 > prepared operation key=TQRZT37AB9WC tag=7CQIVWAOK72C type=del size=3179163 2023-12-03T11:30:48.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.541+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.0 size=1048576 2023-12-03T11:30:48.543 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.542+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.1 size=1048576 2023-12-03T11:30:48.544 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.543+0000 7fc553ded780 1 < completed part key=_multipart_8GRA2MUBGWJI.2~NE4HE1E0CDOC.2 size=439612 2023-12-03T11:30:48.546 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.545+0000 7fc553ded780 1 < completed multipart upload key=8GRA2MUBGWJI upload=NE4HE1E0CDOC tag=XCWCLK7MY8HM type=write size=3585340 2023-12-03T11:30:48.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.547+0000 7fc553ded780 1 > prepared operation key=G1OBBD75M5ZQ tag=3Y9NRI2716GZ type=write size=1031241 2023-12-03T11:30:48.550 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.549+0000 7fc553ded780 1 < suggested remove operation key=WDV4CW863V33 tag=UC51SO6VGA1O type=del 2023-12-03T11:30:48.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.551+0000 7fc553ded780 1 > prepared operation key=ICDF1EYNI5IU tag=351CU23E1J7N type=del size=122181 2023-12-03T11:30:48.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.552+0000 7fc553ded780 1 < completed part key=_multipart_706CET5PSDEG.2~M2ZLZSSO2UT9.0 size=292631 2023-12-03T11:30:48.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.554+0000 7fc553ded780 1 < completed multipart upload key=706CET5PSDEG upload=M2ZLZSSO2UT9 tag=TNJ6O7VT3UTJ type=write size=1341207 2023-12-03T11:30:48.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:30:48.555+0000 7fc553ded780 1 > prepared operation key=WOKG1DISIKK0 tag=4UH16HXRSJ1T type=write size=57913 2023-12-03T11:30:48.557 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] cls_rgw_stats.simulate (9596 ms) 2023-12-03T11:30:48.557 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 1 test from cls_rgw_stats (9597 ms total) 2023-12-03T11:30:48.557 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:30:48.557 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:30:48.917 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 1 test from 1 test suite ran. (11801 ms total) 2023-12-03T11:30:48.918 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 1 test. 2023-12-03T11:30:48.921 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:30:48.922 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:30:48.994 INFO:tasks.workunit:Running workunits matching cls/test_cls_cmpomap.sh on client.0... 2023-12-03T11:30:48.996 INFO:tasks.workunit:Running workunit cls/test_cls_cmpomap.sh... 2023-12-03T11:30:48.996 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_cmpomap.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_cmpomap.sh 2023-12-03T11:30:49.065 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:30:49.066 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 31 tests from 1 test suite. 2023-12-03T11:30:49.066 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:30:50.931 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 31 tests from CmpOmap 2023-12-03T11:30:50.931 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str 2023-12-03T11:30:50.948 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str (17 ms) 2023-12-03T11:30:50.948 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_str_default 2023-12-03T11:30:50.952 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_noexist_str_default (5 ms) 2023-12-03T11:30:50.952 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64 2023-12-03T11:30:50.957 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64 (3 ms) 2023-12-03T11:30:50.957 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_noexist_u64_default 2023-12-03T11:30:50.960 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_noexist_u64_default (4 ms) 2023-12-03T11:30:50.960 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_str 2023-12-03T11:30:50.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_str (7 ms) 2023-12-03T11:30:50.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_u64 2023-12-03T11:30:50.972 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_u64 (6 ms) 2023-12-03T11:30:50.972 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_input 2023-12-03T11:30:50.977 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_input (5 ms) 2023-12-03T11:30:50.977 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_default 2023-12-03T11:30:50.981 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_default (4 ms) 2023-12-03T11:30:50.981 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_u64_empty_default 2023-12-03T11:30:50.985 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_u64_empty_default (4 ms) 2023-12-03T11:30:50.985 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_u64_invalid_value 2023-12-03T11:30:50.989 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_u64_invalid_value (4 ms) 2023-12-03T11:30:50.990 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_at_max_keys 2023-12-03T11:30:50.990 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_at_max_keys (1 ms) 2023-12-03T11:30:50.990 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_vals_over_max_keys 2023-12-03T11:30:50.991 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_vals_over_max_keys (1 ms) 2023-12-03T11:30:50.991 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str 2023-12-03T11:30:50.997 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str (6 ms) 2023-12-03T11:30:50.997 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_str_default 2023-12-03T11:30:51.005 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_str_default (7 ms) 2023-12-03T11:30:51.005 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64 2023-12-03T11:30:51.011 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64 (7 ms) 2023-12-03T11:30:51.011 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_noexist_u64_default 2023-12-03T11:30:51.018 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_noexist_u64_default (7 ms) 2023-12-03T11:30:51.018 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_str 2023-12-03T11:30:51.026 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_str (8 ms) 2023-12-03T11:30:51.026 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64 2023-12-03T11:30:51.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_u64 (9 ms) 2023-12-03T11:30:51.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_einval 2023-12-03T11:30:51.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_einval (3 ms) 2023-12-03T11:30:51.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_u64_eio 2023-12-03T11:30:51.040 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_u64_eio (2 ms) 2023-12-03T11:30:51.040 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_at_max_keys 2023-12-03T11:30:51.041 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_at_max_keys (1 ms) 2023-12-03T11:30:51.041 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_set_vals_over_max_keys 2023-12-03T11:30:51.042 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_set_vals_over_max_keys (1 ms) 2023-12-03T11:30:51.042 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_str 2023-12-03T11:30:51.049 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_str (7 ms) 2023-12-03T11:30:51.049 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_noexist_u64 2023-12-03T11:30:51.055 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_noexist_u64 (6 ms) 2023-12-03T11:30:51.055 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_str 2023-12-03T11:30:51.062 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_str (7 ms) 2023-12-03T11:30:51.063 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64 2023-12-03T11:30:51.071 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64 (9 ms) 2023-12-03T11:30:51.071 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_einval 2023-12-03T11:30:51.073 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_einval (2 ms) 2023-12-03T11:30:51.073 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_eio 2023-12-03T11:30:51.075 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_eio (2 ms) 2023-12-03T11:30:51.075 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_at_max_keys 2023-12-03T11:30:51.076 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_at_max_keys (1 ms) 2023-12-03T11:30:51.076 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_over_max_keys 2023-12-03T11:30:51.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_over_max_keys (1 ms) 2023-12-03T11:30:51.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] CmpOmap.cmp_rm_keys_u64_empty 2023-12-03T11:30:51.084 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] CmpOmap.cmp_rm_keys_u64_empty (7 ms) 2023-12-03T11:30:51.085 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 31 tests from CmpOmap (154 ms total) 2023-12-03T11:30:51.085 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:30:51.085 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:30:51.938 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 31 tests from 1 test suite ran. (2873 ms total) 2023-12-03T11:30:51.939 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 31 tests. 2023-12-03T11:30:51.942 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:30:51.942 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:30:52.018 INFO:tasks.workunit:Running workunits matching cls/test_cls_2pc_queue.sh on client.0... 2023-12-03T11:30:52.019 INFO:tasks.workunit:Running workunit cls/test_cls_2pc_queue.sh... 2023-12-03T11:30:52.020 DEBUG:teuthology.orchestra.run.smithi012:workunit test cls/test_cls_2pc_queue.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_2pc_queue.sh 2023-12-03T11:30:52.096 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:30:52.096 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:30:52.096 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:30:52.096 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from TestCls2PCQueue 2023-12-03T11:30:52.096 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.GetCapacity 2023-12-03T11:30:54.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.GetCapacity (2869 ms) 2023-12-03T11:30:54.966 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.AsyncGetCapacity 2023-12-03T11:30:57.986 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.AsyncGetCapacity (3021 ms) 2023-12-03T11:30:57.986 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.Reserve 2023-12-03T11:31:01.010 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.Reserve (3023 ms) 2023-12-03T11:31:01.010 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.AsyncReserve 2023-12-03T11:31:04.025 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.AsyncReserve (3015 ms) 2023-12-03T11:31:04.025 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.Commit 2023-12-03T11:31:07.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.Commit (3012 ms) 2023-12-03T11:31:07.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.Stats 2023-12-03T11:31:10.061 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.Stats (3021 ms) 2023-12-03T11:31:10.061 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.UpgradeFromReef 2023-12-03T11:31:14.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.UpgradeFromReef (4018 ms) 2023-12-03T11:31:14.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.Abort 2023-12-03T11:31:17.114 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.Abort (3036 ms) 2023-12-03T11:31:17.114 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.ReserveError 2023-12-03T11:31:20.124 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.ReserveError (3010 ms) 2023-12-03T11:31:20.124 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.CommitError 2023-12-03T11:31:23.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.CommitError (3036 ms) 2023-12-03T11:31:23.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.AbortError 2023-12-03T11:31:26.172 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.AbortError (3013 ms) 2023-12-03T11:31:26.172 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.MultiReserve 2023-12-03T11:31:29.194 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.MultiReserve (3022 ms) 2023-12-03T11:31:29.194 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.MultiCommit 2023-12-03T11:31:32.196 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.MultiCommit (3002 ms) 2023-12-03T11:31:32.197 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.MultiAbort 2023-12-03T11:31:35.243 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.MultiAbort (3046 ms) 2023-12-03T11:31:35.243 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.ReserveCommit 2023-12-03T11:31:38.267 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.ReserveCommit (3024 ms) 2023-12-03T11:31:38.267 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.ReserveAbort 2023-12-03T11:31:41.275 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.ReserveAbort (3009 ms) 2023-12-03T11:31:41.276 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.ManualCleanup 2023-12-03T11:31:44.307 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.ManualCleanup (3032 ms) 2023-12-03T11:31:44.307 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.Cleanup 2023-12-03T11:31:47.333 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.Cleanup (3025 ms) 2023-12-03T11:31:47.333 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.MultiProducer 2023-12-03T11:31:57.938 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.MultiProducer (10603 ms) 2023-12-03T11:31:57.938 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.AsyncConsumer 2023-12-03T11:32:01.975 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.AsyncConsumer (4037 ms) 2023-12-03T11:32:01.975 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.MultiProducerConsumer 2023-12-03T11:32:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:Queue was never full - all reservations were successful.Please decrease the amount of consumer threads 2023-12-03T11:32:12.943 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.MultiProducerConsumer (10968 ms) 2023-12-03T11:32:12.943 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.ReserveSpillover 2023-12-03T11:32:16.974 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.ReserveSpillover (4031 ms) 2023-12-03T11:32:16.975 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.CommitSpillover 2023-12-03T11:32:22.998 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.CommitSpillover (6023 ms) 2023-12-03T11:32:22.998 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestCls2PCQueue.AbortSpillover 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestCls2PCQueue.AbortSpillover (5043 ms) 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from TestCls2PCQueue (95944 ms total) 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 24 tests from 1 test suite ran. (95944 ms total) 2023-12-03T11:32:28.041 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 24 tests. 2023-12-03T11:32:28.043 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:32:28.044 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:32:28.119 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_gc_log.sh on client.0... 2023-12-03T11:32:28.120 INFO:tasks.workunit:Running workunit rgw/test_rgw_gc_log.sh... 2023-12-03T11:32:28.120 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/test_rgw_gc_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_gc_log.sh 2023-12-03T11:32:28.200 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:28.200 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:28.201 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 6 tests from 1 test suite. 2023-12-03T11:32:28.201 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:30.054 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 6 tests from rgw_gc_log 2023-12-03T11:32:30.054 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.init_existing_queue 2023-12-03T11:32:30.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.init_existing_queue (24 ms) 2023-12-03T11:32:30.077 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.init_existing_omap 2023-12-03T11:32:30.081 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.init_existing_omap (4 ms) 2023-12-03T11:32:30.081 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.enqueue1_after_init 2023-12-03T11:32:30.083 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.enqueue1_after_init (2 ms) 2023-12-03T11:32:30.083 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.enqueue2_before_init 2023-12-03T11:32:30.085 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.enqueue2_before_init (2 ms) 2023-12-03T11:32:30.085 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.defer1_after_init 2023-12-03T11:32:30.088 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.defer1_after_init (3 ms) 2023-12-03T11:32:30.088 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] rgw_gc_log.defer2_before_init 2023-12-03T11:32:30.089 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] rgw_gc_log.defer2_before_init (1 ms) 2023-12-03T11:32:30.089 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 6 tests from rgw_gc_log (36 ms total) 2023-12-03T11:32:30.089 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:30.089 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:31.069 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 6 tests from 1 test suite ran. (2868 ms total) 2023-12-03T11:32:31.069 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 6 tests. 2023-12-03T11:32:31.076 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:32:31.076 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:32:31.151 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_obj.sh on client.0... 2023-12-03T11:32:31.152 INFO:tasks.workunit:Running workunit rgw/test_rgw_obj.sh... 2023-12-03T11:32:31.152 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/test_rgw_obj.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_obj.sh 2023-12-03T11:32:31.237 INFO:tasks.workunit.client.0.smithi012.stdout:Running main() from gmock_main.cc 2023-12-03T11:32:31.237 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 4 tests from 1 test suite. 2023-12-03T11:32:31.237 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:31.237 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 4 tests from TestRGWObj 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestRGWObj.underscore 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i= 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.238 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i= 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i= 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.241 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.242 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.243 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.244 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.245 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.246 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i= 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i= 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i= 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.247 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.248 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.249 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.250 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.251 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.252 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i=v1 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i=v1 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns= i=v1 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.253 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.254 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.255 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.256 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.257 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.258 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:32:31.259 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: _obj[v1] 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: _obj ns=ns i=v1 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.260 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.261 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.262 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.263 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_obj", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.264 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestRGWObj.underscore (0 ms) 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestRGWObj.no_underscore 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i= 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i= 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i= 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.265 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.266 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.267 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.268 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.269 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i= 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i= 2023-12-03T11:32:31.270 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i= 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.271 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.272 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.273 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.274 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.275 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i=v1 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i=v1 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns= i=v1 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.276 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.277 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.278 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.279 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.280 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:32:31.281 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout:parsed: obj[v1] 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout:stripped: obj ns=ns i=v1 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "obj1": { 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.282 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "obj2": { 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.283 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3": { 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.284 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "obj3-2": { 2023-12-03T11:32:31.285 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "obj", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "v1", 2023-12-03T11:32:31.286 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestRGWObj.no_underscore (0 ms) 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestRGWObj.obj_to_raw 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.287 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.288 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.289 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker_myobj", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.290 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.291 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.292 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker_myobj", 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.293 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.294 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.295 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.296 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.297 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.298 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.299 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.300 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.301 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.302 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.303 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.304 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.305 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.306 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.307 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.308 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.309 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.310 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.311 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.312 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.313 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.314 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.315 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.316 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.317 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.318 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj", 2023-12-03T11:32:31.319 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.320 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.321 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.322 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj", 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.323 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.324 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.325 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.326 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.327 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.328 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.329 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.330 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.331 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.332 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.333 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.334 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.335 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.336 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.337 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.338 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.339 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.340 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.341 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.342 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.343 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.344 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.345 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.346 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.347 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.348 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.349 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.350 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj_", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.351 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.352 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.353 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj_", 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.354 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.355 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.356 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.357 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:32:31.358 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.359 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.360 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.361 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:32:31.362 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.363 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.364 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.365 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:32:31.366 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.367 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.368 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.369 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.370 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.371 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.372 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.373 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": "default-placement.data", 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.374 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "test", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": "", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": "" 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.375 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "obj": { 2023-12-03T11:32:31.376 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.377 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.378 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.379 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestRGWObj.obj_to_raw (1 ms) 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] TestRGWObj.old_to_raw 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker_myobj", 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.380 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker_myobj 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.381 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker_myobj", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.382 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker_myobj 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.383 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.384 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns_myobj 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns_myobj", 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.385 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.386 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns_myobj 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.387 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__:inst_myobj 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.388 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst_myobj", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.389 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__:inst_myobj 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:32:31.390 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.391 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns:inst_myobj 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst_myobj", 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.392 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "myobj", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.393 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns:inst_myobj 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj", 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.394 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker___myobj 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.395 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.396 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker___myobj 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.397 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.398 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns__myobj 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj", 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.399 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.400 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns__myobj 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.401 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__:inst__myobj 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.402 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj" 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.403 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__:inst__myobj 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.404 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.405 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns:inst__myobj 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj", 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.406 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.407 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns:inst__myobj 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj_", 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.408 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.409 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker___myobj_ 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker___myobj_", 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.410 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker___myobj_ 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.411 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.412 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns__myobj_ 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.413 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns__myobj_", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.414 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "", 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns__myobj_ 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.415 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.416 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__:inst__myobj_ 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__:inst__myobj_", 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "marker__myobj_" 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.417 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.418 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "" 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__:inst__myobj_ 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj": { 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj": { 2023-12-03T11:32:31.419 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.420 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout:raw=.data-pool:marker__ns:inst__myobj_ 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "raw_obj2": { 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "pool": ".data-pool", 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "oid": "marker__ns:inst__myobj_", 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "loc": "" 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "new_obj2": { 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket": { 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "ebtest", 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "marker": "marker", 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_id": "bucket-id", 2023-12-03T11:32:31.421 INFO:tasks.workunit.client.0.smithi012.stdout: "tenant": "", 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "explicit_placement": { 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "data_pool": ".data-pool", 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "data_extra_pool": "", 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "index_pool": ".index-pool" 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "key": { 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "name": "_myobj_", 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "instance": "inst", 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: "ns": "ns" 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.422 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:raw2=.data-pool:marker__ns:inst__myobj_ 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] TestRGWObj.old_to_raw (0 ms) 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 4 tests from TestRGWObj (1 ms total) 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 4 tests from 1 test suite ran. (1 ms total) 2023-12-03T11:32:31.423 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 4 tests. 2023-12-03T11:32:31.423 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:32:31.423 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:32:31.458 INFO:tasks.workunit:Running workunits matching rgw/test_librgw_file.sh on client.0... 2023-12-03T11:32:31.460 INFO:tasks.workunit:Running workunit rgw/test_librgw_file.sh... 2023-12-03T11:32:31.460 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/test_librgw_file.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh 2023-12-03T11:32:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:31.588 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:31.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.602+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 realm 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.604+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.605+0000 7fe8c2111800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:32:31.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.605+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.606+0000 7fe8c2111800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:32:31.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.606+0000 7fe8c2111800 20 searching for the correct realm 2023-12-03T11:32:31.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.618+0000 7fe8bfc18640 0 -- 172.21.15.12:0/3248578207 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a48dcb8080 msgr2=0x55a48dcba470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:32:31.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:32:31.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:32:31.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:32:31.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:32:31.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.619+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:32:31.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:32:31.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 zone default found 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 4 Realm: () 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 zonegroup default 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.620+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.621+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.621+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.621+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.621+0000 7fe8c2111800 20 started sync module instance, tier type = 2023-12-03T11:32:31.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.621+0000 7fe8c2111800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:32:31.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.630+0000 7fe8c0419640 0 -- 172.21.15.12:0/3248578207 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55a48dcc0110 msgr2=0x55a48dcc25d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=0 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=1 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=2 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=3 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=4 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=5 2023-12-03T11:32:31.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.632+0000 7fe8c2111800 20 add_watcher() i=6 2023-12-03T11:32:31.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.633+0000 7fe8c2111800 20 add_watcher() i=7 2023-12-03T11:32:31.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.633+0000 7fe8c2111800 2 all 8 watchers are set, enabling cache 2023-12-03T11:32:31.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.634+0000 7fe885fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:32:31.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.635+0000 7fe8c2111800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:32:31.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.635+0000 7fe8c2111800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:32:31.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.641+0000 7fe8c2111800 5 note: GC not initialized 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe7fcff9640 20 reqs_thread_entry: start 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe8c2111800 20 init_complete bucket index max shards: 1 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe8c2111800 20 Filter name: none 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe7f27fc640 20 reqs_thread_entry: start 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe8c2111800 10 cache get: name=default.rgw.meta+users.uid+ceph-test-librgw-file : miss 2023-12-03T11:32:31.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.642+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x0 2023-12-03T11:32:31.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 adding default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache get: name=default.rgw.meta+users.email+librgw@example.com : miss 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 adding default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache get: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= : miss 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 20 rados->read ofs=0 len=0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= info.flags=0x0 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 adding default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= to cache LRU end 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache get: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= : hit (negative entry) 2023-12-03T11:32:31.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.643+0000 7fe8c2111800 10 cache get: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= : hit (negative entry) 2023-12-03T11:32:31.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.645+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-12-03T11:32:31.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.645+0000 7fe8c2111800 10 moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:32:31.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.645+0000 7fe8c2111800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:ceph-test-librgw-file, ofs0, ns] 2023-12-03T11:32:31.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.645+0000 7fe8a67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1047972020225 cookie 94165241971312 notifier 5628 bl.length()=663 2023-12-03T11:32:31.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.646+0000 7fe8a67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ceph-test-librgw-file info.flags=0x17 2023-12-03T11:32:31.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.646+0000 7fe8a67fc640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ceph-test-librgw-file to cache LRU end 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.647+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.647+0000 7fe8c2111800 10 moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.647+0000 7fe8c2111800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:users.email:librgw@example.com, ofs0, ns] 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.648+0000 7fe8a67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1047972020224 cookie 94165241929984 notifier 5628 bl.length()=194 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.648+0000 7fe8a67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+librgw@example.com info.flags=0x7 2023-12-03T11:32:31.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.648+0000 7fe8a67fc640 10 rgw watcher librados: moving default.rgw.meta+users.email+librgw@example.com to cache LRU end 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.649+0000 7fe8c2111800 10 cache put: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= info.flags=0x7 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.649+0000 7fe8c2111800 10 moving default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= to cache LRU end 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.649+0000 7fe8c2111800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:oLf1t8+zeWnIBQSummRepwYJqDA=, ofs0, ns] 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.650+0000 7fe8a67fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1047972020224 cookie 94165241984784 notifier 5628 bl.length()=203 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.650+0000 7fe8a67fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= info.flags=0x7 2023-12-03T11:32:31.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.650+0000 7fe8a67fc640 10 rgw watcher librados: moving default.rgw.meta+users.keys+oLf1t8+zeWnIBQSummRepwYJqDA= to cache LRU end 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "user_id": "ceph-test-librgw-file", 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "display_name": "librgw test user", 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "email": "librgw@example.com", 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "suspended": 0, 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "max_buckets": 1000, 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "subusers": [], 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "keys": [ 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: { 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "user": "ceph-test-librgw-file", 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "access_key": "oLf1t8+zeWnIBQSummRepwYJqDA=", 2023-12-03T11:32:31.652 INFO:tasks.workunit.client.0.smithi012.stdout: "secret_key": "vMlRbII+kwUojKNrXxx3TiS2/SqetYYT4vsiP+8PNMYs/6ZlwU+ZHQ==" 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "swift_keys": [], 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "caps": [], 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "op_mask": "read, write, delete", 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "default_placement": "", 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "default_storage_class": "", 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "placement_tags": [], 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_quota": { 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "enabled": false, 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size": -1, 2023-12-03T11:32:31.653 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "max_objects": -1 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "user_quota": { 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "enabled": false, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size": -1, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "max_objects": -1 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "temp_url_keys": [], 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "type": "rgw", 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout: "mfa_ids": [] 2023-12-03T11:32:31.654 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:32:31.655 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:31.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.651+0000 7fe885fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:32:31.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.654+0000 7fe8c2111800 20 remove_watcher() i=0 2023-12-03T11:32:31.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.654+0000 7fe8c2111800 2 removed watcher, disabling cache 2023-12-03T11:32:31.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.655+0000 7fe8c2111800 20 remove_watcher() i=1 2023-12-03T11:32:31.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.656+0000 7fe8c2111800 20 remove_watcher() i=2 2023-12-03T11:32:31.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.657+0000 7fe8c2111800 20 remove_watcher() i=3 2023-12-03T11:32:31.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.658+0000 7fe8c2111800 20 remove_watcher() i=4 2023-12-03T11:32:31.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.659+0000 7fe8c2111800 20 remove_watcher() i=5 2023-12-03T11:32:31.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.660+0000 7fe8c2111800 20 remove_watcher() i=6 2023-12-03T11:32:31.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:31.661+0000 7fe8c2111800 20 remove_watcher() i=7 2023-12-03T11:32:31.667 INFO:tasks.workunit.client.0.smithi012.stdout:phase 1.1 2023-12-03T11:32:31.687 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:32:31.687 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:31.687 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:32:31.687 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:31.697 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:31.697 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:32.071 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (385 ms) 2023-12-03T11:32:32.071 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:32:32.072 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT_NOROOT (1 ms) 2023-12-03T11:32:32.072 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:32.072 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:32:32.072 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:32:32.080 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/quux 2023-12-03T11:32:32.085 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/f1 2023-12-03T11:32:32.090 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/f2 2023-12-03T11:32:32.093 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/f1 2023-12-03T11:32:32.096 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/d1/ 2023-12-03T11:32:32.100 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/hungry 2023-12-03T11:32:32.104 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/hungry/ 2023-12-03T11:32:32.108 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/momma 2023-12-03T11:32:32.113 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/bear/ 2023-12-03T11:32:32.116 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/sasquatch 2023-12-03T11:32:32.119 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/sasquatch/ 2023-12-03T11:32:32.122 INFO:tasks.workunit.client.0.smithi012.stdout:creating: nfsroot:foo/bar/baz/frobozz 2023-12-03T11:32:32.125 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_HIER1 (54 ms) 2023-12-03T11:32:32.126 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:32:32.230 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_DIRS1 (104 ms) 2023-12-03T11:32:32.231 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:32:32.231 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:32:32.231 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:32:32.248 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (18 ms) 2023-12-03T11:32:32.248 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:32:32.286 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (38 ms) 2023-12-03T11:32:32.286 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:32:32.286 INFO:tasks.workunit.client.0.smithi012.stdout:rename file rfile_0 to rfile_r0 (bucket brename0) 2023-12-03T11:32:32.293 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (7 ms) 2023-12-03T11:32:32.293 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:32:32.293 INFO:tasks.workunit.client.0.smithi012.stdout:rename file rfile_1 (bucket brename0) to rfile_rhilldog (bucket brename1) 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (8 ms) 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_0 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_1 2023-12-03T11:32:32.301 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_2 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_0 2023-12-03T11:32:32.302 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_0/sfile_0 2023-12-03T11:32:32.303 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:32.303 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_0/sfile_1 2023-12-03T11:32:32.304 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:32.305 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_1 2023-12-03T11:32:32.305 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_1/sfile_0 2023-12-03T11:32:32.306 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:32.306 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_1/sfile_1 2023-12-03T11:32:32.308 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:32.308 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_2 2023-12-03T11:32:32.308 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_2/sfile_0 2023-12-03T11:32:32.309 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:32.309 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_2/sfile_1 2023-12-03T11:32:32.310 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:32.311 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READ_DIRS1 (9 ms) 2023-12-03T11:32:32.311 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:32:32.491 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_SETUP1 (180 ms) 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:32.492 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:32.493 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-12-03T11:32:32.493 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:32:32.498 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/f2 (f2): FILE 2023-12-03T11:32:32.498 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/f1 (f1): FILE 2023-12-03T11:32:32.501 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-12-03T11:32:32.502 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-12-03T11:32:32.505 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:32:32.505 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:32:32.506 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-12-03T11:32:32.506 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-12-03T11:32:32.507 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:32:32.507 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:32:32.508 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar (bar): DIR 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo (foo): DIR 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot (nfsroot): DIR 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.HIER1 (18 ms) 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:32:32.510 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:32.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:32.509+0000 7fc8b98128c0 -1 shutting down 2023-12-03T11:32:32.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:32.510+0000 7fc890593640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 96985, si_uid : 1000, si_addr0x3e800017ad9, si_status0 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (119 ms) 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW (942 ms total) 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 24 tests from 1 test suite ran. (942 ms total) 2023-12-03T11:32:32.629 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 24 tests. 2023-12-03T11:32:32.634 INFO:tasks.workunit.client.0.smithi012.stdout:phase 1.2 2023-12-03T11:32:32.655 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:32:32.655 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:32.655 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:32:32.655 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:32.663 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:32.663 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:33.034 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (380 ms) 2023-12-03T11:32:33.034 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:32:33.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT_NOROOT (1 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_HIER1 (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_DIRS1 (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:32:33.036 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READ_DIRS1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:32:33.037 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.HIER1 (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:33.038 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:33.039 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:33.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.034+0000 7f0c56d6a8c0 -1 shutting down 2023-12-03T11:32:33.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.035+0000 7f0c2ff17640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97095, si_uid : 1000, si_addr0x3e800017b47, si_status0 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (64 ms) 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW (445 ms total) 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 24 tests from 1 test suite ran. (445 ms total) 2023-12-03T11:32:33.099 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 24 tests. 2023-12-03T11:32:33.104 INFO:tasks.workunit.client.0.smithi012.stdout:phase 1.3 2023-12-03T11:32:33.124 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 24 tests from 1 test suite. 2023-12-03T11:32:33.124 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:33.125 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW 2023-12-03T11:32:33.125 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:33.134 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:33.134 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:33.298 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (173 ms) 2023-12-03T11:32:33.298 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT_NOROOT 2023-12-03T11:32:33.298 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT_NOROOT (1 ms) 2023-12-03T11:32:33.298 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:33.299 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:32:33.299 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_HIER1 2023-12-03T11:32:33.303 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_HIER1 (5 ms) 2023-12-03T11:32:33.303 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETUP_DIRS1 2023-12-03T11:32:33.365 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETUP_DIRS1 (62 ms) 2023-12-03T11:32:33.365 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SETATTR 2023-12-03T11:32:33.365 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SETATTR (0 ms) 2023-12-03T11:32:33.365 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CREATE_DIRS1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CREATE_DIRS1 (0 ms) 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_SETUP_RENAME1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_SETUP_RENAME1 (0 ms) 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_INTRABUCKET_RENAME1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_INTRABUCKET_RENAME1 (0 ms) 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RGW_CROSSBUCKET_RENAME1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RGW_CROSSBUCKET_RENAME1 (0 ms) 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.GETATTR_DIRS1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_0 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_0/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:33.366 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_1/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout:scanning objects in /bdirs1/dir_2 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sdir_0 (sdir_0): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sfile_0 (sfile_0): FILE mode: 33412 size: 16 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sdir_1 (sdir_1): DIR mode: 17139 nlinks: 1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout: /bdirs1/dir_2/sfile_1 (sfile_1): FILE mode: 33412 size: 16 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.GETATTR_DIRS1 (0 ms) 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READ_DIRS1 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_0 2023-12-03T11:32:33.367 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_0/sfile_0 2023-12-03T11:32:33.368 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:33.368 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_0/sfile_1 2023-12-03T11:32:33.368 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:33.368 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_1 2023-12-03T11:32:33.368 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_1/sfile_0 2023-12-03T11:32:33.370 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:33.370 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_1/sfile_1 2023-12-03T11:32:33.371 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:33.371 INFO:tasks.workunit.client.0.smithi012.stdout:read back objects in /bdirs1/dir_2 2023-12-03T11:32:33.371 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_2/sfile_0 2023-12-03T11:32:33.372 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_0 : "data for sfile_0" 2023-12-03T11:32:33.372 INFO:tasks.workunit.client.0.smithi012.stdout:reading 0,256 dir_2/sfile_1 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout: read back from sfile_1 : "data for sfile_1" 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READ_DIRS1 (8 ms) 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_SETUP1 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_SETUP1 (0 ms) 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READF_DIRS1 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READF_DIRS1 (0 ms) 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITEF_DIRS1 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITEF_DIRS1 (0 ms) 2023-12-03T11:32:33.374 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.RELEASE_DIRS1 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_0 type: DIR refs: 2 2023-12-03T11:32:33.375 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_0 type: FILE refs: 2 2023-12-03T11:32:33.376 INFO:tasks.workunit.client.0.smithi012.stdout:release sdir_1 type: DIR refs: 2 2023-12-03T11:32:33.376 INFO:tasks.workunit.client.0.smithi012.stdout:release sfile_1 type: FILE refs: 2 2023-12-03T11:32:33.376 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.RELEASE_DIRS1 (1 ms) 2023-12-03T11:32:33.376 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.HIER1 2023-12-03T11:32:33.379 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/f2 (f2): FILE 2023-12-03T11:32:33.380 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/f1 (f1): FILE 2023-12-03T11:32:33.382 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/f1 (f1): FILE 2023-12-03T11:32:33.383 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/d1 (d1): DIR 2023-12-03T11:32:33.386 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:32:33.386 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/sasquatch (sasquatch): FILE 2023-12-03T11:32:33.386 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/quux (quux): FILE 2023-12-03T11:32:33.387 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/momma (momma): FILE 2023-12-03T11:32:33.387 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:32:33.388 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/hungry (hungry): FILE 2023-12-03T11:32:33.388 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/frobozz (frobozz): FILE 2023-12-03T11:32:33.390 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz/bear (bear): DIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar/baz (baz): DIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo/bar (bar): DIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot/foo (foo): DIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:/nfsroot (nfsroot): DIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.HIER1 (16 ms) 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (0 ms) 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:32:33.391 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.389+0000 7faa288688c0 -1 shutting down 2023-12-03T11:32:33.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.390+0000 7faa03749640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97205, si_uid : 1000, si_addr0x3e800017bb5, si_status0 2023-12-03T11:32:33.463 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (73 ms) 2023-12-03T11:32:33.464 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 24 tests from LibRGW (339 ms total) 2023-12-03T11:32:33.464 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:33.464 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:33.464 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 24 tests from 1 test suite ran. (339 ms total) 2023-12-03T11:32:33.464 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 24 tests. 2023-12-03T11:32:33.469 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2.1 2023-12-03T11:32:33.489 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:32:33.489 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:33.490 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from LibRGW 2023-12-03T11:32:33.490 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:33.498 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:33.498 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:33.659 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (170 ms) 2023-12-03T11:32:33.660 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:33.660 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:33.660 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:33.668 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (8 ms) 2023-12-03T11:32:33.668 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:33.668 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:32:33.669 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-12-03T11:32:33.676 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave0 2023-12-03T11:32:33.683 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave1 2023-12-03T11:32:33.690 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave2 2023-12-03T11:32:33.697 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave3 2023-12-03T11:32:33.704 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave4 2023-12-03T11:32:33.711 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave5 2023-12-03T11:32:33.718 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave6 2023-12-03T11:32:33.725 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave7 2023-12-03T11:32:33.732 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave8 2023-12-03T11:32:33.738 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave9 2023-12-03T11:32:33.738 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (70 ms) 2023-12-03T11:32:33.738 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (146 ms) 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:33.885 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:33.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.883+0000 7f4b4b52f8c0 -1 shutting down 2023-12-03T11:32:33.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:33.884+0000 7f4b24560640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97315, si_uid : 1000, si_addr0x3e800017c23, si_status0 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (41 ms) 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from LibRGW (436 ms total) 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 9 tests from 1 test suite ran. (436 ms total) 2023-12-03T11:32:33.925 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 9 tests. 2023-12-03T11:32:33.930 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2.2 2023-12-03T11:32:33.951 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 9 tests from 1 test suite. 2023-12-03T11:32:33.951 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:33.951 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from LibRGW 2023-12-03T11:32:33.951 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:33.960 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:33.960 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (169 ms) 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (0 ms) 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:32:34.121 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:34.140 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (18 ms) 2023-12-03T11:32:34.140 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET_MULTI 2023-12-03T11:32:34.147 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave0 2023-12-03T11:32:34.154 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave1 2023-12-03T11:32:34.161 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave2 2023-12-03T11:32:34.168 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave3 2023-12-03T11:32:34.174 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave4 2023-12-03T11:32:34.182 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave5 2023-12-03T11:32:34.188 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave6 2023-12-03T11:32:34.195 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave7 2023-12-03T11:32:34.202 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave8 2023-12-03T11:32:34.209 INFO:tasks.workunit.client.0.smithi012.stdout:created: sorrydave9 2023-12-03T11:32:34.209 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET_MULTI (70 ms) 2023-12-03T11:32:34.209 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET_MULTI 2023-12-03T11:32:34.362 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET_MULTI (153 ms) 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:34.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:34.361+0000 7f8e736c58c0 -1 shutting down 2023-12-03T11:32:34.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:34.363+0000 7f8e4bfee640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97425, si_uid : 1000, si_addr0x3e800017c91, si_status0 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (124 ms) 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 9 tests from LibRGW (535 ms total) 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 9 tests from 1 test suite ran. (535 ms total) 2023-12-03T11:32:34.487 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 9 tests. 2023-12-03T11:32:34.492 INFO:tasks.workunit.client.0.smithi012.stdout:phase 3.1 2023-12-03T11:32:34.816 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 13 tests from 1 test suite. 2023-12-03T11:32:34.817 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:34.817 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 13 tests from LibRGW 2023-12-03T11:32:34.817 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:34.825 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:34.825 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:34.996 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (179 ms) 2023-12-03T11:32:34.996 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:34.997 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (2 ms) 2023-12-03T11:32:34.997 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:35.004 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (7 ms) 2023-12-03T11:32:35.004 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:32:35.007 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-12-03T11:32:35.007 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:32:35.009 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (2 ms) 2023-12-03T11:32:35.009 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LARGE1 2023-12-03T11:32:35.064 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LARGE1 (55 ms) 2023-12-03T11:32:35.064 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LARGE2 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LARGE2 (43 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:35.108 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:35.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:35.106+0000 7f5dcf3228c0 -1 shutting down 2023-12-03T11:32:35.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:35.108+0000 7f5da8341640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97535, si_uid : 1000, si_addr0x3e800017cff, si_status0 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (53 ms) 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 13 tests from LibRGW (344 ms total) 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 13 tests from 1 test suite ran. (344 ms total) 2023-12-03T11:32:35.160 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 13 tests. 2023-12-03T11:32:35.166 INFO:tasks.workunit.client.0.smithi012.stdout:phase 3.2 2023-12-03T11:32:35.493 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 13 tests from 1 test suite. 2023-12-03T11:32:35.493 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:35.493 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 13 tests from LibRGW 2023-12-03T11:32:35.493 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:35.501 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:35.665 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (172 ms) 2023-12-03T11:32:35.665 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:35.666 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:35.666 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:35.666 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:32:35.667 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:32:35.670 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (4 ms) 2023-12-03T11:32:35.670 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:32:35.674 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (4 ms) 2023-12-03T11:32:35.674 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LARGE1 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LARGE1 (53 ms) 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LARGE2 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LARGE2 (0 ms) 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:32:35.727 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:32:35.732 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-12-03T11:32:35.732 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:35.745 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (14 ms) 2023-12-03T11:32:35.745 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:35.745 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:35.746 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:35.746 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:35.746 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:35.744+0000 7fc182b218c0 -1 shutting down 2023-12-03T11:32:35.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:35.745+0000 7fc15bb76640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97645, si_uid : 1000, si_addr0x3e800017d6d, si_status0 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (85 ms) 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 13 tests from LibRGW (337 ms total) 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 13 tests from 1 test suite ran. (337 ms total) 2023-12-03T11:32:35.830 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 13 tests. 2023-12-03T11:32:35.836 INFO:tasks.workunit.client.0.smithi012.stdout:phase 4.1 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from LibRGW 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.TVAR 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:read i1: 64001 2023-12-03T11:32:35.857 INFO:tasks.workunit.client.0.smithi012.stdout:read s1: blunderbuss 2023-12-03T11:32:35.858 INFO:tasks.workunit.client.0.smithi012.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-12-03T11:32:35.858 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:35.868 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:35.868 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:36.020 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (162 ms) 2023-12-03T11:32:36.020 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:36.020 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:36.020 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:32:36.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (15 ms) 2023-12-03T11:32:36.035 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_0 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_1 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_2 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_3 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_4 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_5 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_6 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_7 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_8 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_9 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_10 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_11 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_12 2023-12-03T11:32:36.619 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_13 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_14 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_15 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_16 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_17 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_18 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_19 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_20 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_21 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_22 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_23 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_24 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_25 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_26 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_27 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_28 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_29 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_30 2023-12-03T11:32:36.620 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_31 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_32 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_33 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_34 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_35 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_36 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_37 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_38 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_39 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_40 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_41 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_42 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_43 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_44 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_45 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_46 2023-12-03T11:32:36.621 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_47 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_48 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_49 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_50 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_51 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_52 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_53 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_54 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_55 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_56 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_57 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_58 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_59 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_60 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_61 2023-12-03T11:32:36.622 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_62 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_63 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_64 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_65 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_66 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_67 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_68 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_69 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_70 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_71 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_72 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_73 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_74 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_75 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_76 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_77 2023-12-03T11:32:36.623 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_78 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_79 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_80 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_81 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_82 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_83 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_84 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_85 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_86 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_87 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_88 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_89 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_90 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_91 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_92 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_93 2023-12-03T11:32:36.624 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_94 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_95 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_96 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_97 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_98 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:created: dmarker:f_99 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (583 ms) 2023-12-03T11:32:36.625 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=. flags=4 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=.. flags=4 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:32:36.626 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:32:36.627 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:32:36.628 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:32:36.629 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:32:36.630 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:32:36.631 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:32:36.632 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_READDIR (3 ms) 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:32:36.633 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:32:36.634 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:32:36.635 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:32:36.636 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:32:36.637 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:32:36.638 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:32:36.639 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER2_READDIR (2 ms) 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:36.640 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:36.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:36.623+0000 7fb9e661c8c0 -1 shutting down 2023-12-03T11:32:36.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:36.624+0000 7fb9bdeb5640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97755, si_uid : 1000, si_addr0x3e800017ddb, si_status0 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (61 ms) 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from LibRGW (827 ms total) 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 11 tests from 1 test suite ran. (827 ms total) 2023-12-03T11:32:36.684 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 11 tests. 2023-12-03T11:32:36.689 INFO:tasks.workunit.client.0.smithi012.stdout:phase 4.2 2023-12-03T11:32:36.709 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 11 tests from 1 test suite. 2023-12-03T11:32:36.709 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:36.709 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from LibRGW 2023-12-03T11:32:36.709 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.TVAR 2023-12-03T11:32:36.710 INFO:tasks.workunit.client.0.smithi012.stdout:read i1: 64001 2023-12-03T11:32:36.710 INFO:tasks.workunit.client.0.smithi012.stdout:read s1: blunderbuss 2023-12-03T11:32:36.710 INFO:tasks.workunit.client.0.smithi012.stdout:read s3: [ OK ] LibRGW.TVAR (0 ms) 2023-12-03T11:32:36.710 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:36.717 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:36.717 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:36.868 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (159 ms) 2023-12-03T11:32:36.868 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:36.869 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:36.869 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_BUCKET 2023-12-03T11:32:36.877 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_BUCKET (8 ms) 2023-12-03T11:32:36.877 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_SETUP_OBJECTS 2023-12-03T11:32:36.877 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_SETUP_OBJECTS (0 ms) 2023-12-03T11:32:36.878 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_READDIR 2023-12-03T11:32:36.878 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_READDIR (0 ms) 2023-12-03T11:32:36.878 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER2_READDIR 2023-12-03T11:32:36.880 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_0 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_1 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_10 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_11 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_12 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_13 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_14 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_15 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_16 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_17 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_18 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_19 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_2 flags=8 2023-12-03T11:32:36.881 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_20 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_21 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_22 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_23 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_24 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_25 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_26 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_27 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_28 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_29 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_3 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_30 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_31 flags=8 2023-12-03T11:32:36.882 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_32 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_33 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_34 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_35 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_36 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_37 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_38 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_39 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_4 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_40 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_41 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_42 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_43 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_44 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_45 flags=8 2023-12-03T11:32:36.883 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_46 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_47 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_48 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_49 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_5 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_50 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_51 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_52 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_53 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_54 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_55 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_56 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_57 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_58 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_59 flags=8 2023-12-03T11:32:36.884 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_6 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_60 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_61 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_62 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_63 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_64 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_65 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_66 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_67 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_68 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_69 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_7 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_70 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_71 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_72 flags=8 2023-12-03T11:32:36.885 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_73 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_74 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_75 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_76 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_77 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_78 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_79 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_8 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_80 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_81 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_82 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_83 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_84 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_85 flags=8 2023-12-03T11:32:36.886 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_86 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_87 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_88 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_89 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_9 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_90 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_91 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_92 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_93 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_94 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_95 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_96 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_97 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_98 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:r2_cb bucket=dmarker dir=nfs_marker iv count=0 called back name=f_99 flags=8 2023-12-03T11:32:36.887 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER2_READDIR (3 ms) 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MARKER1_OBJ_CLEANUP 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MARKER1_OBJ_CLEANUP (0 ms) 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:36.879+0000 7fe30df5d8c0 -1 shutting down 2023-12-03T11:32:36.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:36.880+0000 7fe2e7104640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97865, si_uid : 1000, si_addr0x3e800017e49, si_status0 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (52 ms) 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 11 tests from LibRGW (223 ms total) 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 11 tests from 1 test suite ran. (223 ms total) 2023-12-03T11:32:36.932 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 11 tests. 2023-12-03T11:32:36.938 INFO:tasks.workunit.client.0.smithi012.stdout:phase 5.1 2023-12-03T11:32:36.968 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 19 tests from 1 test suite. 2023-12-03T11:32:36.969 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:36.969 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 19 tests from LibRGW 2023-12-03T11:32:36.969 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:36.979 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:36.981 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:37.142 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (173 ms) 2023-12-03T11:32:37.142 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:37.143 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:37.143 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:37.150 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (7 ms) 2023-12-03T11:32:37.150 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:32:37.153 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (3 ms) 2023-12-03T11:32:37.153 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-12-03T11:32:37.153 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-12-03T11:32:37.153 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:32:37.154 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (1 ms) 2023-12-03T11:32:37.154 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-12-03T11:32:37.154 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-12-03T11:32:37.155 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-12-03T11:32:37.159 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.PUT_OBJECT (5 ms) 2023-12-03T11:32:37.159 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.GET_OBJECT (2 ms) 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITEV 2023-12-03T11:32:37.161 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READV 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READV (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_OBJECT (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:37.162 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:37.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:37.160+0000 7f2d14ebf8c0 -1 shutting down 2023-12-03T11:32:37.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:37.161+0000 7f2ced90b640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 97975, si_uid : 1000, si_addr0x3e800017eb7, si_status0 2023-12-03T11:32:37.206 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (45 ms) 2023-12-03T11:32:37.206 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 19 tests from LibRGW (238 ms total) 2023-12-03T11:32:37.206 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:37.206 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:37.207 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 19 tests from 1 test suite ran. (238 ms total) 2023-12-03T11:32:37.207 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 19 tests. 2023-12-03T11:32:37.212 INFO:tasks.workunit.client.0.smithi012.stdout:phase 5.2 2023-12-03T11:32:37.241 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] Running 19 tests from 1 test suite. 2023-12-03T11:32:37.241 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment set-up. 2023-12-03T11:32:37.241 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 19 tests from LibRGW 2023-12-03T11:32:37.241 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.INIT 2023-12-03T11:32:37.248 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:32:37.248 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:32:37.410 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.INIT (169 ms) 2023-12-03T11:32:37.410 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.MOUNT 2023-12-03T11:32:37.411 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.MOUNT (1 ms) 2023-12-03T11:32:37.411 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CREATE_BUCKET 2023-12-03T11:32:37.411 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CREATE_BUCKET (0 ms) 2023-12-03T11:32:37.411 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_BUCKET 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_BUCKET (5 ms) 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LIST_OBJECTS 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LIST_OBJECTS (0 ms) 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.LOOKUP_OBJECT 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.LOOKUP_OBJECT (0 ms) 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.OBJ_OPEN 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.OBJ_OPEN (0 ms) 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.PUT_OBJECT 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.PUT_OBJECT (0 ms) 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.GET_OBJECT 2023-12-03T11:32:37.417 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.GET_OBJECT (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.STAT_OBJECT 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.STAT_OBJECT (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITE_READ_VERIFY 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITE_READ_VERIFY (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.WRITEV 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.WRITEV (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READV 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READV (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.READV_AFTER_WRITEV 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.READV_AFTER_WRITEV (0 ms) 2023-12-03T11:32:37.418 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_OBJECT 2023-12-03T11:32:37.421 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_OBJECT (4 ms) 2023-12-03T11:32:37.421 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.DELETE_BUCKET 2023-12-03T11:32:37.436 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.DELETE_BUCKET (15 ms) 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.CLEANUP 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.CLEANUP (0 ms) 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.UMOUNT 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.UMOUNT (0 ms) 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stdout:[ RUN ] LibRGW.SHUTDOWN 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:37.435+0000 7f88f02448c0 -1 shutting down 2023-12-03T11:32:37.437 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:32:37.436+0000 7f88cb125640 -1 received signal: User defined signal 1, si_code : -6, si_value (int): 0, si_value (ptr): 0, si_errno: 0, si_pid : 98085, si_uid : 1000, si_addr0x3e800017f25, si_status0 2023-12-03T11:32:37.475 INFO:tasks.workunit.client.0.smithi012.stdout:[ OK ] LibRGW.SHUTDOWN (39 ms) 2023-12-03T11:32:37.475 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] 19 tests from LibRGW (234 ms total) 2023-12-03T11:32:37.475 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:32:37.475 INFO:tasks.workunit.client.0.smithi012.stdout:[----------] Global test environment tear-down 2023-12-03T11:32:37.476 INFO:tasks.workunit.client.0.smithi012.stdout:[==========] 19 tests from 1 test suite ran. (234 ms total) 2023-12-03T11:32:37.476 INFO:tasks.workunit.client.0.smithi012.stdout:[ PASSED ] 19 tests. 2023-12-03T11:32:37.481 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:32:37.482 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:32:37.513 INFO:tasks.workunit:Stopping ['cls/test_cls_lock.sh', 'cls/test_cls_log.sh', 'cls/test_cls_refcount.sh', 'cls/test_cls_rgw.sh', 'cls/test_cls_rgw_gc.sh', 'cls/test_cls_rgw_stats.sh', 'cls/test_cls_cmpomap.sh', 'cls/test_cls_2pc_queue.sh', 'rgw/test_rgw_gc_log.sh', 'rgw/test_rgw_obj.sh', 'rgw/test_librgw_file.sh'] on client.0... 2023-12-03T11:32:37.513 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:32:37.836 DEBUG:teuthology.parallel:result is None 2023-12-03T11:32:37.837 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:32:37.867 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:32:37.867 DEBUG:teuthology.orchestra.run.smithi012:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:32:37.923 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:32:37.923 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:32:37.933 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:32:37.933 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:32:37.933 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:32:37.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:32:37.979 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:32:37.979 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:32:38.034 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:32:38.034 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:32:38.089 INFO:tasks.workunit:timeout=3h 2023-12-03T11:32:38.089 INFO:tasks.workunit:cleanup=True 2023-12-03T11:32:38.089 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:32:38.145 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:33:44.856 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 95% (11565/12123) Updating files: 96% (11639/12123) Updating files: 97% (11760/12123) Updating files: 98% (11881/12123) Updating files: 99% (12002/12123) Updating files: 100% (12123/12123) Updating files: 100% (12123/12123), done. 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:33:45.503 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:33:45.504 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:33:45.506 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:33:45.561 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:33:45.564 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:33:45.564 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:33:45.614 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:33:45.665 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:33:45.705 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:33:45.707 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:33:45.707 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:33:45.747 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:33:45.750 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:33:45.751 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:33:45.807 INFO:tasks.workunit:Running workunits matching rgw/test_rgw_s3_mp_reupload.sh on client.0... 2023-12-03T11:33:45.808 INFO:tasks.workunit:Running workunit rgw/test_rgw_s3_mp_reupload.sh... 2023-12-03T11:33:45.808 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/test_rgw_s3_mp_reupload.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_s3_mp_reupload.sh 2023-12-03T11:33:45.868 INFO:tasks.workunit.client.0.smithi012.stdout:INFO: fully qualified domain name: smithi012.front.sepia.ceph.com 2023-12-03T11:33:45.868 INFO:tasks.workunit.client.0.smithi012.stdout:== TESTING *ABORTING* MULTIPART UPLOAD WITH RE-UPLOADS == 2023-12-03T11:33:45.868 INFO:tasks.workunit.client.0.smithi012.stdout:== TESTING NON-VERSIONED BUCKET == 2023-12-03T11:33:48.556 INFO:tasks.workunit.client.0.smithi012.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:33:48.767 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting pip 2023-12-03T11:33:48.773 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:33:48.823 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: pip 2023-12-03T11:33:48.823 INFO:tasks.workunit.client.0.smithi012.stdout: Attempting uninstall: pip 2023-12-03T11:33:48.823 INFO:tasks.workunit.client.0.smithi012.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:33:48.983 INFO:tasks.workunit.client.0.smithi012.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:33:48.988 INFO:tasks.workunit.client.0.smithi012.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:33:50.010 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed pip-23.3.1 2023-12-03T11:33:50.912 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting boto3 2023-12-03T11:33:50.913 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:33:51.515 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:33:51.516 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:33:51.576 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:33:51.577 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:33:51.639 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:33:51.640 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:33:51.706 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:33:51.707 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:33:51.790 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:33:51.792 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:33:51.955 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:33:51.956 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:33:51.981 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:33:51.982 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:33:52.005 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:33:52.006 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:33:52.110 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:33:53.263 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:33:53.485 INFO:tasks.workunit.client.0.smithi012.stdout:INFO: creating user 2023-12-03T11:33:53.578 INFO:tasks.workunit.client.0.smithi012.stdout:{ 2023-12-03T11:33:53.578 INFO:tasks.workunit.client.0.smithi012.stdout: "user_id": "testid", 2023-12-03T11:33:53.578 INFO:tasks.workunit.client.0.smithi012.stdout: "display_name": "M. Tester", 2023-12-03T11:33:53.578 INFO:tasks.workunit.client.0.smithi012.stdout: "email": "tester@ceph.com", 2023-12-03T11:33:53.578 INFO:tasks.workunit.client.0.smithi012.stdout: "suspended": 0, 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "max_buckets": 1000, 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "subusers": [], 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "keys": [ 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: { 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "user": "testid", 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "access_key": "0555b35654ad1656d804", 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==" 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: } 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: ], 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "swift_keys": [], 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "caps": [], 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "op_mask": "read, write, delete", 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "default_placement": "", 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "default_storage_class": "", 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "placement_tags": [], 2023-12-03T11:33:53.579 INFO:tasks.workunit.client.0.smithi012.stdout: "bucket_quota": { 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "enabled": false, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "user_quota": { 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "enabled": false, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: }, 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "temp_url_keys": [], 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "type": "rgw", 2023-12-03T11:33:53.580 INFO:tasks.workunit.client.0.smithi012.stdout: "mfa_ids": [] 2023-12-03T11:33:53.581 INFO:tasks.workunit.client.0.smithi012.stdout:} 2023-12-03T11:33:53.581 INFO:tasks.workunit.client.0.smithi012.stdout: 2023-12-03T11:33:53.828 INFO:tasks.workunit.client.0.smithi012.stderr:5+0 records in 2023-12-03T11:33:53.828 INFO:tasks.workunit.client.0.smithi012.stderr:5+0 records out 2023-12-03T11:33:53.828 INFO:tasks.workunit.client.0.smithi012.stderr:5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.0141428 s, 371 MB/s 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:bucket nams is reupload-bkt-118463 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:endpoint is http://smithi012.front.sepia.ceph.com:80 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:start UploadId=2~_1x84nwX7P-SKznQA5HWBdKtKjJ_Jd4 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 1 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 1 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 2} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.064 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:phase 2 uploaded part {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1} 2023-12-03T11:33:56.065 INFO:tasks.workunit.client.0.smithi012.stdout:aborting multipart upload, parts=[{'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 1}, {'ETag': '"bc961e51a4b083d1b9696c5fa03224aa"', 'PartNumber': 2}] 2023-12-03T11:33:56.599 INFO:tasks.workunit.client.0.smithi012.stdout:radosgw-admin bucket radoslist: 2023-12-03T11:33:56.659 INFO:tasks.workunit.client.0.smithi012.stdout:rados ls: 2023-12-03T11:33:56.671 INFO:tasks.workunit.client.0.smithi012.stdout:== PASSED == 2023-12-03T11:33:56.672 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:33:56.672 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:33:56.743 INFO:tasks.workunit:Stopping ['rgw/test_rgw_s3_mp_reupload.sh'] on client.0... 2023-12-03T11:33:56.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:33:57.101 DEBUG:teuthology.parallel:result is None 2023-12-03T11:33:57.102 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:33:57.132 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:33:57.132 DEBUG:teuthology.orchestra.run.smithi012:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:33:57.188 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:33:57.188 INFO:teuthology.run_tasks:Running task ragweed... 2023-12-03T11:33:57.206 DEBUG:tasks.ragweed:ragweed config is {'client.0': {'default-branch': 'ceph-master', 'rgw_server': 'client.0', 'stages': 'prepare,check'}} 2023-12-03T11:33:57.206 INFO:tasks.ragweed:Downloading ragweed... 2023-12-03T11:33:57.206 INFO:tasks.ragweed:Using branch 'wip-alimasa-multi-pubsub-etag' for ragweed 2023-12-03T11:33:57.207 DEBUG:teuthology.orchestra.run.smithi012:> git clone -b wip-alimasa-multi-pubsub-etag https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:33:57.243 INFO:teuthology.orchestra.run.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2023-12-03T11:33:57.289 INFO:teuthology.orchestra.run.smithi012.stderr:fatal: Remote branch wip-alimasa-multi-pubsub-etag not found in upstream origin 2023-12-03T11:33:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-12-03T11:33:57.291 INFO:tasks.ragweed:Using branch 'ceph-master' for ragweed 2023-12-03T11:33:57.291 DEBUG:teuthology.orchestra.run.smithi012:> git clone -b ceph-master https://git.ceph.com/ragweed.git /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:33:57.346 INFO:teuthology.orchestra.run.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/ragweed.client.0'... 2023-12-03T11:33:57.478 INFO:tasks.ragweed:Creating rgw users... 2023-12-03T11:33:57.479 DEBUG:tasks.ragweed:Creating user ragweed.client.0 on client.0 2023-12-03T11:33:57.479 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid ragweed.client.0 --display-name 'Mr. ragweed.client.0' --access-key GQAYCLCOVNHYCDNTYBWK --secret YcsPZDHXZS+P5k/r+/WQJa0Oe1exxGuzi5Zju7hBcoSM5pKU6vekHg== --email ragweed.client.0+test@test.test --system false 2023-12-03T11:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:33:57.572 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:33:57.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.582+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 realm 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.586 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.584+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.586+0000 7f3061337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:33:57.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.586+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.587+0000 7f3061337800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:33:57.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.587+0000 7f3061337800 20 searching for the correct realm 2023-12-03T11:33:57.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.595+0000 7f305d0cd640 0 -- 172.21.15.12:0/3542213742 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f305401f3f0 msgr2=0x7f305401f870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.598+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:33:57.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:33:57.601 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 zone default found 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 4 Realm: () 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 zonegroup default 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.599+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.600+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.600+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.600+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.600+0000 7f3061337800 20 started sync module instance, tier type = 2023-12-03T11:33:57.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.600+0000 7f3061337800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:33:57.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.601+0000 7f305d8ce640 0 -- 172.21.15.12:0/3542213742 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f3054042950 msgr2=0x7f3054017420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=0 2023-12-03T11:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=1 2023-12-03T11:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=2 2023-12-03T11:33:57.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=3 2023-12-03T11:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=4 2023-12-03T11:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=5 2023-12-03T11:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.609+0000 7f3061337800 20 add_watcher() i=6 2023-12-03T11:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.610+0000 7f3061337800 20 add_watcher() i=7 2023-12-03T11:33:57.611 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.610+0000 7f3061337800 2 all 8 watchers are set, enabling cache 2023-12-03T11:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.612+0000 7f3022fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.612+0000 7f3061337800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:33:57.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.612+0000 7f3061337800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f3061337800 5 note: GC not initialized 2023-12-03T11:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f2f91ffb640 20 reqs_thread_entry: start 2023-12-03T11:33:57.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f3061337800 20 init_complete bucket index max shards: 1 2023-12-03T11:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f3061337800 20 Filter name: none 2023-12-03T11:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f2f837fe640 20 reqs_thread_entry: start 2023-12-03T11:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f3061337800 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2023-12-03T11:33:57.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.617+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 cache get: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test : miss 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 adding default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 10 cache get: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK : miss 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.618+0000 7f3061337800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.619+0000 7f3061337800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.620 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.619+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK info.flags=0x0 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.619+0000 7f3061337800 10 adding default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK to cache LRU end 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.619+0000 7f3061337800 10 cache get: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK : hit (negative entry) 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.619+0000 7f3061337800 10 cache get: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK : hit (negative entry) 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f3061337800 10 moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:33:57.621 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f3061337800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2023-12-03T11:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f303f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922131 cookie 94052690233216 notifier 5991 bl.length()=641 2023-12-03T11:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f303f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x17 2023-12-03T11:33:57.622 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.620+0000 7f303f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:33:57.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.622+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-12-03T11:33:57.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.622+0000 7f3061337800 10 moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.623 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.622+0000 7f3061337800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2023-12-03T11:33:57.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.623+0000 7f303f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922167 cookie 94052690221232 notifier 5991 bl.length()=202 2023-12-03T11:33:57.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.623+0000 7f303f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+ragweed.client.0+test@test.test info.flags=0x7 2023-12-03T11:33:57.624 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.623+0000 7f303f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+ragweed.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.624+0000 7f3061337800 10 cache put: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK info.flags=0x7 2023-12-03T11:33:57.625 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.624+0000 7f3061337800 10 moving default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK to cache LRU end 2023-12-03T11:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.624+0000 7f3061337800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:GQAYCLCOVNHYCDNTYBWK, ofs0, ns] 2023-12-03T11:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.625+0000 7f303effd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922132 cookie 94052690233216 notifier 5991 bl.length()=190 2023-12-03T11:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.625+0000 7f303effd640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK info.flags=0x7 2023-12-03T11:33:57.626 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.625+0000 7f303effd640 10 rgw watcher librados: moving default.rgw.meta+users.keys+GQAYCLCOVNHYCDNTYBWK to cache LRU end 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "user_id": "ragweed.client.0", 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "display_name": "Mr. ragweed.client.0", 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "email": "ragweed.client.0+test@test.test", 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "suspended": 0, 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "max_buckets": 1000, 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "subusers": [], 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "keys": [ 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: { 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "user": "ragweed.client.0", 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "GQAYCLCOVNHYCDNTYBWK", 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "YcsPZDHXZS+P5k/r+/WQJa0Oe1exxGuzi5Zju7hBcoSM5pKU6vekHg==" 2023-12-03T11:33:57.627 INFO:teuthology.orchestra.run.smithi012.stdout: } 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "swift_keys": [], 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": [], 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "op_mask": "read, write, delete", 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "", 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "default_storage_class": "", 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_tags": [], 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:57.628 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "temp_url_keys": [], 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "rgw", 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: "mfa_ids": [] 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-03T11:33:57.629 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:33:57.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.626+0000 7f3022fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:33:57.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.628+0000 7f3061337800 20 remove_watcher() i=0 2023-12-03T11:33:57.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.628+0000 7f3061337800 2 removed watcher, disabling cache 2023-12-03T11:33:57.630 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.629+0000 7f3061337800 20 remove_watcher() i=1 2023-12-03T11:33:57.631 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.630+0000 7f3061337800 20 remove_watcher() i=2 2023-12-03T11:33:57.633 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.632+0000 7f3061337800 20 remove_watcher() i=3 2023-12-03T11:33:57.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.633+0000 7f3061337800 20 remove_watcher() i=4 2023-12-03T11:33:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.634+0000 7f3061337800 20 remove_watcher() i=5 2023-12-03T11:33:57.635 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.635+0000 7f3061337800 20 remove_watcher() i=6 2023-12-03T11:33:57.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.635+0000 7f3061337800 20 remove_watcher() i=7 2023-12-03T11:33:57.642 DEBUG:tasks.ragweed:Creating user sysuser.client.0 on client.0 2023-12-03T11:33:57.642 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid sysuser.client.0 --display-name 'Mr. sysuser.client.0' --access-key JNQIGRCXNWFWPGZXDSSI --secret Mumq8eT3TFc8QQIR1uKclYI4MRZFEtbkiG1A6jiK/dTtSnXjufRELQ== --email sysuser.client.0+test@test.test --system true 2023-12-03T11:33:57.693 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:33:57.694 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:33:57.705 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.704+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 realm 2023-12-03T11:33:57.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.707 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:33:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.706+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.707+0000 7fe83bd11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:33:57.708 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.707+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.708+0000 7fe83bd11800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:33:57.709 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.708+0000 7fe83bd11800 20 searching for the correct realm 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83a012640 0 -- 172.21.15.12:0/1290014785 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x55eb66f85e10 msgr2=0x55eb66f88200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:33:57.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.719+0000 7fe83bd11800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 zone default found 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 4 Realm: () 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:33:57.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 zonegroup default 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 started sync module instance, tier type = 2023-12-03T11:33:57.723 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.720+0000 7fe83bd11800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:33:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=0 2023-12-03T11:33:57.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=1 2023-12-03T11:33:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=2 2023-12-03T11:33:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=3 2023-12-03T11:33:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=4 2023-12-03T11:33:57.731 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.729+0000 7fe83bd11800 20 add_watcher() i=5 2023-12-03T11:33:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.730+0000 7fe83a012640 0 -- 172.21.15.12:0/1290014785 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55eb66f92080 msgr2=0x55eb66f94540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:33:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.730+0000 7fe83bd11800 20 add_watcher() i=6 2023-12-03T11:33:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.730+0000 7fe83bd11800 20 add_watcher() i=7 2023-12-03T11:33:57.732 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.730+0000 7fe83bd11800 2 all 8 watchers are set, enabling cache 2023-12-03T11:33:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.732+0000 7fe7ff7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:33:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.732+0000 7fe83bd11800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:33:57.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.732+0000 7fe83bd11800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe83bd11800 5 note: GC not initialized 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe76e7fc640 20 reqs_thread_entry: start 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe83bd11800 20 init_complete bucket index max shards: 1 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe83bd11800 20 Filter name: none 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe75ffff640 20 reqs_thread_entry: start 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.737+0000 7fe83bd11800 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2023-12-03T11:33:57.739 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x0 2023-12-03T11:33:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:33:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 cache get: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test : miss 2023-12-03T11:33:57.740 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 adding default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 cache get: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI : miss 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados->read ofs=0 len=0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI info.flags=0x0 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.738+0000 7fe83bd11800 10 adding default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI to cache LRU end 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.739+0000 7fe83bd11800 10 cache get: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI : hit (negative entry) 2023-12-03T11:33:57.741 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.739+0000 7fe83bd11800 10 cache get: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI : hit (negative entry) 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe83bd11800 10 moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe83bd11800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe81bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922168 cookie 94469533297440 notifier 6006 bl.length()=641 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe81bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x17 2023-12-03T11:33:57.742 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.740+0000 7fe81bfff640 10 rgw watcher librados: moving default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:33:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-12-03T11:33:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe83bd11800 10 moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.743 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe83bd11800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2023-12-03T11:33:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe81bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922133 cookie 94469533301184 notifier 6006 bl.length()=202 2023-12-03T11:33:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe81bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+sysuser.client.0+test@test.test info.flags=0x7 2023-12-03T11:33:57.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.742+0000 7fe81bfff640 10 rgw watcher librados: moving default.rgw.meta+users.email+sysuser.client.0+test@test.test to cache LRU end 2023-12-03T11:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe83bd11800 10 cache put: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI info.flags=0x7 2023-12-03T11:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe83bd11800 10 moving default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI to cache LRU end 2023-12-03T11:33:57.745 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe83bd11800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.keys:JNQIGRCXNWFWPGZXDSSI, ofs0, ns] 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe81bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1060856922126 cookie 94469533298480 notifier 6006 bl.length()=190 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe81bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI info.flags=0x7 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.744+0000 7fe81bfff640 10 rgw watcher librados: moving default.rgw.meta+users.keys+JNQIGRCXNWFWPGZXDSSI to cache LRU end 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout: "user_id": "sysuser.client.0", 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout: "display_name": "Mr. sysuser.client.0", 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout: "email": "sysuser.client.0+test@test.test", 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout: "suspended": 0, 2023-12-03T11:33:57.746 INFO:teuthology.orchestra.run.smithi012.stdout: "max_buckets": 1000, 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "subusers": [], 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "keys": [ 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: { 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "user": "sysuser.client.0", 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "JNQIGRCXNWFWPGZXDSSI", 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "Mumq8eT3TFc8QQIR1uKclYI4MRZFEtbkiG1A6jiK/dTtSnXjufRELQ==" 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: } 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "swift_keys": [], 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": [], 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "op_mask": "read, write, delete", 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "system": true, 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "", 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "default_storage_class": "", 2023-12-03T11:33:57.747 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_tags": [], 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:33:57.748 INFO:teuthology.orchestra.run.smithi012.stdout: "temp_url_keys": [], 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "rgw", 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stdout: "mfa_ids": [] 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.745+0000 7fe7ff7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.748+0000 7fe83bd11800 20 remove_watcher() i=0 2023-12-03T11:33:57.749 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.748+0000 7fe83bd11800 2 removed watcher, disabling cache 2023-12-03T11:33:57.750 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.749+0000 7fe83bd11800 20 remove_watcher() i=1 2023-12-03T11:33:57.751 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.750+0000 7fe83bd11800 20 remove_watcher() i=2 2023-12-03T11:33:57.752 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.751+0000 7fe83bd11800 20 remove_watcher() i=3 2023-12-03T11:33:57.753 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.752+0000 7fe83bd11800 20 remove_watcher() i=4 2023-12-03T11:33:57.754 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.753+0000 7fe83bd11800 20 remove_watcher() i=5 2023-12-03T11:33:57.755 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.754+0000 7fe83bd11800 20 remove_watcher() i=6 2023-12-03T11:33:57.756 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:33:57.755+0000 7fe83bd11800 20 remove_watcher() i=7 2023-12-03T11:33:57.761 INFO:tasks.ragweed:Configuring ragweed... 2023-12-03T11:33:57.761 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:33:57.762 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/archive/ragweed.client.0.conf 2023-12-03T11:33:57.777 INFO:tasks.ragweed:Configuring boto... 2023-12-03T11:33:57.777 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:33:57.778 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/boto.cfg 2023-12-03T11:33:57.832 DEBUG:teuthology.orchestra.run.smithi012:ragweed tests against rgw> source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/ragweed.client.0 && RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg tox --sitepackages -- -v -m 'not fails_on_rgw' 2023-12-03T11:33:58.005 INFO:teuthology.orchestra.run.smithi012.stdout:GLOB sdist-make: /home/ubuntu/cephtest/ragweed.client.0/setup.py 2023-12-03T11:33:58.174 INFO:teuthology.orchestra.run.smithi012.stdout:py create: /home/ubuntu/cephtest/ragweed.client.0/.tox/py 2023-12-03T11:33:58.660 INFO:teuthology.orchestra.run.smithi012.stdout:py installdeps: -rrequirements.txt 2023-12-03T11:34:24.609 INFO:teuthology.orchestra.run.smithi012.stdout:py inst: /home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip 2023-12-03T11:34:27.765 INFO:teuthology.orchestra.run.smithi012.stdout:py installed: asyncssh==2.13.2,Babel==2.9.1,bcrypt==3.2.2,boto==2.49.0,cachetools==4.2.4,ceph==1.0.0,ceph-volume==1.0.0,cephfs==2.0.0,certifi==2021.10.8,cffi==1.14.5,chardet==4.0.0,cheroot==8.6.0,CherryPy==18.6.1,cockpit @ file:///builddir/build/BUILD/cockpit-299/tmp/wheel/cockpit-299-py3-none-any.whl,cryptography==36.0.1,dasbus==1.4,dbus-python==1.2.18,distro==1.5.0,docopt==0.6.2,exceptiongroup==1.2.0,execnet==2.0.2,file-magic==0.4.0,gevent==23.9.1,google-auth==2.23.4,gpg==1.15.1,greenlet==3.0.1,httplib2==0.22.0,idna==2.10,iniconfig==2.0.0,isodate==0.6.1,jaraco.classes==3.2.1,jaraco.collections==3.0.0,jaraco.functools==3.5.0,jaraco.text==3.2.0,Jinja2==2.11.3,jsonpatch==1.21,jsonpointer==2.0,kubernetes==26.1.0,libcomps==0.1.18,logutils==0.3.5,lxml==4.6.5,Mako==1.1.4.dev0,MarkupSafe==1.1.1,more-itertools==8.12.0,munch==4.0.0,natsort==7.1.1,nftables==0.1,numpy==1.20.1,oauthlib==3.1.1,packaging==23.2,pecan==1.4.2,perf==0.1,pexpect==4.8.0,pluggy==1.3.0,ply==3.11,portend==3.1.0,prettytable==0.7.2,psutil==5.8.0,ptyprocess==0.6.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.20,PyGObject==3.40.1,PyJWT==2.4.0,pyOpenSSL==21.0.0,pyparsing==2.4.7,PySocks==1.7.1,pytest==7.4.3,python-dateutil==2.8.1,python-linux-procfs==0.7.0,pytz==2021.1,pyudev==0.22.0,PyYAML==5.4.1,rados==2.0.0,ragweed @ file:///home/ubuntu/cephtest/ragweed.client.0/.tox/.tmp/package/1/ragweed-0.0.1.zip#sha256=ab16d96a174d9765d4ef8c0f775332c3aba85d1a934b3e8c913bdb2cf776e75c,rbd==2.0.0,registries-conf-ctl==1.0,remoto==1.2.1,repoze.lru==0.7,requests==2.25.1,requests-oauthlib==1.3.0,rgw==2.0.0,Routes==2.5.1,rpm==4.16.1.3,rsa==4.9,s3cmd==2.3.0,scipy==1.6.2,selinux==3.5,sepolicy==3.5,setools==4.4.3,setroubleshoot @ file:///builddir/build/BUILD/setroubleshoot-3.3.32/src,six==1.15.0,sos==4.3,systemd-python==234,tempora==5.0.0,toml==0.10.2,tomli==2.0.1,typing_extensions==4.4.0,urllib3==1.26.5,WebOb==1.8.7,websocket-client==1.2.3,Werkzeug==2.0.3,zc.lockfile==2.0,zope.event==5.0,zope.interface==6.1 2023-12-03T11:34:27.766 INFO:teuthology.orchestra.run.smithi012.stdout:py run-test-pre: PYTHONHASHSEED='4042765855' 2023-12-03T11:34:27.766 INFO:teuthology.orchestra.run.smithi012.stdout:py run-test: commands[0] | python -m pytest -v -m 'not fails_on_rgw' 2023-12-03T11:34:27.987 INFO:teuthology.orchestra.run.smithi012.stdout:============================= test session starts ============================== 2023-12-03T11:34:27.987 INFO:teuthology.orchestra.run.smithi012.stdout:platform linux -- Python 3.9.17, pytest-7.4.3, pluggy-1.3.0 -- /home/ubuntu/cephtest/ragweed.client.0/.tox/py/bin/python 2023-12-03T11:34:27.987 INFO:teuthology.orchestra.run.smithi012.stdout:cachedir: .tox/py/.pytest_cache 2023-12-03T11:34:27.987 INFO:teuthology.orchestra.run.smithi012.stdout:rootdir: /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:34:27.987 INFO:teuthology.orchestra.run.smithi012.stdout:configfile: pytest.ini 2023-12-03T11:34:28.091 INFO:teuthology.orchestra.run.smithi012.stdout:collecting ... collected 9 items 2023-12-03T11:34:28.091 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:34:28.570 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_small_obj_data::test PASSED [ 11%] 2023-12-03T11:34:29.069 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_multipart_simple::test PASSED [ 22%] 2023-12-03T11:34:29.502 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_multipart_index_versioning::test PASSED [ 33%] 2023-12-03T11:34:30.038 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_multipart_defer_complete::test PASSED [ 44%] 2023-12-03T11:34:30.576 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_multipart_defer_update_complete::test PASSED [ 55%] 2023-12-03T11:34:37.397 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_obj_storage_class::test PASSED [ 66%] 2023-12-03T11:34:38.645 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart::test PASSED [ 77%] 2023-12-03T11:34:39.812 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_copy::test PASSED [ 88%] 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout:ragweed/tests/tests.py::r_test_obj_storage_class_multipart_copy::test PASSED [100%] 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout:=============================== warnings summary =============================== 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout:.tox/py/lib/python3.9/site-packages/boto/plugin.py:40 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: /home/ubuntu/cephtest/ragweed.client.0/.tox/py/lib/python3.9/site-packages/boto/plugin.py:40: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: import imp 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout:-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 2023-12-03T11:34:41.999 INFO:teuthology.orchestra.run.smithi012.stdout:======================== 9 passed, 1 warning in 14.01s ========================= 2023-12-03T11:34:42.031 INFO:teuthology.orchestra.run.smithi012.stdout:___________________________________ summary ____________________________________ 2023-12-03T11:34:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: py: commands succeeded 2023-12-03T11:34:42.031 INFO:teuthology.orchestra.run.smithi012.stdout: congratulations :) 2023-12-03T11:34:42.046 INFO:tasks.ragweed:Cleaning up boto... 2023-12-03T11:34:42.046 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/boto.cfg 2023-12-03T11:34:42.061 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid ragweed.client.0 --purge-data 2023-12-03T11:34:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:34:42.144 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:34:42.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.154+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 realm 2023-12-03T11:34:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:34:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.158 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.156+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.158+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:34:42.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.158+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.159+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:34:42.160 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.159+0000 7fcdc0321800 20 searching for the correct realm 2023-12-03T11:34:42.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.168+0000 7fcdbc0b8640 0 -- 172.21.15.12:0/1204088100 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7fcdb00143f0 msgr2=0x7fcdb0014850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:34:42.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:34:42.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:34:42.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:34:42.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:34:42.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.170+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 zone default found 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 4 Realm: () 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:34:42.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 zonegroup default 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.171+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.172+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.172+0000 7fcdc0321800 20 started sync module instance, tier type = 2023-12-03T11:34:42.174 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.172+0000 7fcdc0321800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:34:42.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.175+0000 7fcdbb8b7640 0 -- 172.21.15.12:0/1204088100 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x56553f195020 msgr2=0x56553f195480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=0 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=1 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=2 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=3 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=4 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=5 2023-12-03T11:34:42.183 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.182+0000 7fcdc0321800 20 add_watcher() i=6 2023-12-03T11:34:42.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.183+0000 7fcdc0321800 20 add_watcher() i=7 2023-12-03T11:34:42.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.183+0000 7fcdc0321800 2 all 8 watchers are set, enabling cache 2023-12-03T11:34:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.184+0000 7fcd827e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:34:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.185+0000 7fcdc0321800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:34:42.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.185+0000 7fcdc0321800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.227+0000 7fccf17fa640 20 reqs_thread_entry: start 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.227+0000 7fcdc0321800 20 init_complete bucket index max shards: 1 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 20 Filter name: none 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcceaffd640 20 reqs_thread_entry: start 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+users.uid+ragweed.client.0 : miss 2023-12-03T11:34:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=452 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+users.uid+ragweed.client.0 info.flags=0x13 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 10 adding default.rgw.meta+users.uid+ragweed.client.0 to cache LRU end 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.228+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 : miss 2023-12-03T11:34:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 info.flags=0x17 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 to cache LRU end 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.229+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_complete 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_multipart_defer_update_complete 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_multipart_index_versioning 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_multipart_simple 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_copy 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_obj_storage_class_multipart_copy 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 cls_bucket_list_unordered: got tests/r_test_small_obj_data 2023-12-03T11:34:42.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x56553f20e818 s->prefetch_data=0 2023-12-03T11:34:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 10 manifest: total_size = 12870 2023-12-03T11:34:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16126398870454817182 2023-12-03T11:34:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_defer_complete state=0x56553f20e818 s->prefetch_data=0 2023-12-03T11:34:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20e818 2023-12-03T11:34:42.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.230+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=12870 stripe_ofs=12870 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16126398870454817182 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.goh4Tz9xQljybkUMgEfNwI81uf8w6dU_0 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.234 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.233+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16126398870454817182 2023-12-03T11:34:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_complete 2023-12-03T11:34:42.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 10 manifest: total_size = 7558 2023-12-03T11:34:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.664415852394519092 2023-12-03T11:34:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_defer_update_complete state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.236 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.234+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=7558 stripe_ofs=7558 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.664415852394519092 2023-12-03T11:34:42.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.238 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.T_2I9F-xCvgOvUjz1P0-O2ddA_dCaQR_0 2023-12-03T11:34:42.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 327 2023-12-03T11:34:42.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.237+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.664415852394519092 2023-12-03T11:34:42.240 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.239+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_defer_update_complete 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 10 manifest: total_size = 12608 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10713006005016377709 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_index_versioning state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.240+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=12608 stripe_ofs=12608 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10713006005016377709 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.l-yoas34foiVdrDhGQQDauGgYPKQ9y2_0 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.242+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10713006005016377709 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.243+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_index_versioning 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcdc0321800 10 manifest: total_size = 12611 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16701092478190069115 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_multipart_simple state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.244+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=12611 stripe_ofs=12611 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16701092478190069115 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.nHk-WsKr1VbSdaR9ZyUp6pd9FZrj0m__0 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.248 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.246+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16701092478190069115 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.247+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=tests/r_test_multipart_simple 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 10 manifest: total_size = 5146 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2105355681576311583 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.248+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5146 stripe_ofs=5146 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2105355681576311583 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.Btk0UtDG5JLyWnyBODQZBAQVcIjb7qx_0 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.250+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2105355681576311583 2023-12-03T11:34:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.251+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class 2023-12-03T11:34:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 10 manifest: total_size = 5208 2023-12-03T11:34:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17493518590605276871 2023-12-03T11:34:42.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_copy state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.252+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5208 stripe_ofs=5208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17493518590605276871 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.urO1Dy28044OoxneMQJq0QmYSlQyq84_0 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.256 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.255+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17493518590605276871 2023-12-03T11:34:42.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.256+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_copy 2023-12-03T11:34:42.257 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.256+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.257+0000 7fcdc0321800 10 manifest: total_size = 5345 2023-12-03T11:34:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.257+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9707980856756902311 2023-12-03T11:34:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.257+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.257+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.257+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5345 stripe_ofs=5345 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9707980856756902311 2023-12-03T11:34:42.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.260 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.piD16c6KKpEYJ5XUG6P4_4IhG5jCewL_0 2023-12-03T11:34:42.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.259+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9707980856756902311 2023-12-03T11:34:42.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.260+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart 2023-12-03T11:34:42.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.260+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.261+0000 7fcdc0321800 10 manifest: total_size = 5247 2023-12-03T11:34:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.261+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.94119380687097541 2023-12-03T11:34:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.261+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_obj_storage_class_multipart_copy state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.261+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.261+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5247 stripe_ofs=5247 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.94119380687097541 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.x9Z5ci9O6fkApiQxYkQBPY5iruDf95e_0 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 326 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.263+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.94119380687097541 2023-12-03T11:34:42.265 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.264+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=tests/r_test_obj_storage_class_multipart_copy 2023-12-03T11:34:42.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 10 manifest: total_size = 5303 2023-12-03T11:34:42.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18135875262510206899 2023-12-03T11:34:42.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-conf:tests/r_test_small_obj_data state=0x56553f212d18 s->prefetch_data=0 2023-12-03T11:34:42.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f212d18 2023-12-03T11:34:42.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.265+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3.0 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5303 stripe_ofs=5303 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18135875262510206899 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.ZsiZ3VNL7aNpxPx0nBshYajzvAisCbo_0 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.267+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.268+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.14tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18135875262510206899 2023-12-03T11:34:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=tests/r_test_small_obj_data 2023-12-03T11:34:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.271 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.269+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-conf[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-conf : miss 2023-12-03T11:34:42.272 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=204 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-conf info.flags=0x11 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-conf to cache LRU end 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-conf from cache 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.271+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-conf, ofs0, ns] 2023-12-03T11:34:42.273 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.272+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725916 cookie 140521398112256 notifier 6039 bl.length()=162 2023-12-03T11:34:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.273+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 from cache 2023-12-03T11:34:42.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.273+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-conf:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3, ofs0, ns] 2023-12-03T11:34:42.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.274+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725955 cookie 94924128386896 notifier 6039 bl.length()=219 2023-12-03T11:34:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-conf/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 : miss 2023-12-03T11:34:42.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-conf/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 info.flags=0x0 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-conf/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 to cache LRU end 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-conf/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3 from cache 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.276+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-conf/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3, ofs0, ns] 2023-12-03T11:34:42.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.277+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725917 cookie 140521398112256 notifier 6039 bl.length()=216 2023-12-03T11:34:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.279+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 : miss 2023-12-03T11:34:42.280 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.279+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=409 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 info.flags=0x17 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 to cache LRU end 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.280+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.281+0000 7fcdc0321800 10 manifest: total_size = 74198 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.281+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.377449586680940161 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.281+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208180 obj=test-client.0-r-test-multipart-defer-complete.1:foo state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.281+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.282 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.281+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10.0 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4252047 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2023-12-03T11:34:42.285 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.377449586680940161 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.1 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 345 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.2 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 591 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.3 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 837 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.4 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1083 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.286 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.283+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.377449586680940161 2023-12-03T11:34:42.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.284+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:34:42.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.285+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-complete.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.286+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 : miss 2023-12-03T11:34:42.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.286+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=235 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 info.flags=0x11 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 to cache LRU end 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-complete.1 from cache 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-complete.1, ofs0, ns] 2023-12-03T11:34:42.289 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.287+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725956 cookie 94924128386896 notifier 6039 bl.length()=191 2023-12-03T11:34:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.288+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 from cache 2023-12-03T11:34:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.288+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10, ofs0, ns] 2023-12-03T11:34:42.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.289+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725903 cookie 94924130845680 notifier 6039 bl.length()=249 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 : miss 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 info.flags=0x0 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 to cache LRU end 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10 from cache 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.291+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-defer-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10, ofs0, ns] 2023-12-03T11:34:42.293 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.292+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725951 cookie 94924130870832 notifier 6039 bl.length()=246 2023-12-03T11:34:42.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.294+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 : miss 2023-12-03T11:34:42.295 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.294+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=416 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 info.flags=0x17 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 to cache LRU end 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:34:42.296 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x56553f218c38 s->prefetch_data=0 2023-12-03T11:34:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 10 manifest: total_size = 74198 2023-12-03T11:34:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.530211647011875016 2023-12-03T11:34:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-multipart-defer-update-complete.1:foo state=0x56553f218c38 s->prefetch_data=0 2023-12-03T11:34:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218c38 2023-12-03T11:34:42.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.295+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11.0 2023-12-03T11:34:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2023-12-03T11:34:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4252047 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.530211647011875016 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.1 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 345 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.2 2023-12-03T11:34:42.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 591 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.3 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 837 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.4 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1083 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.298+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.7tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.530211647011875016 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.299+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.299+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.299+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.300+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.300+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.300+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.300+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.300+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-defer-update-complete.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 : miss 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=242 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 info.flags=0x11 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 to cache LRU end 2023-12-03T11:34:42.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-defer-update-complete.1 from cache 2023-12-03T11:34:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.301+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-defer-update-complete.1, ofs0, ns] 2023-12-03T11:34:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.302+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725952 cookie 94924130870832 notifier 6039 bl.length()=198 2023-12-03T11:34:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.303+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 from cache 2023-12-03T11:34:42.304 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.303+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-defer-update-complete.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11, ofs0, ns] 2023-12-03T11:34:42.305 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.304+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725953 cookie 94924130870832 notifier 6039 bl.length()=256 2023-12-03T11:34:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 : miss 2023-12-03T11:34:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.307 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 info.flags=0x0 2023-12-03T11:34:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 to cache LRU end 2023-12-03T11:34:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11 from cache 2023-12-03T11:34:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.306+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-defer-update-complete.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11, ofs0, ns] 2023-12-03T11:34:42.308 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.307+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725903 cookie 94924130884384 notifier 6039 bl.length()=253 2023-12-03T11:34:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 : miss 2023-12-03T11:34:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.310 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=409 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 info.flags=0x17 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 to cache LRU end 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.309+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 2023-12-03T11:34:42.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.310+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo[EM0.ypjpco0A03015dk1mmp7pfnDcmx] 2023-12-03T11:34:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.310+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.310+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.311+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to 8n4nqpvryj2ttjn4emik4ldm2pj8b7ul 2023-12-03T11:34:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.311+0000 7fcdc0321800 20 get_obj_state_impl: setting s->olh_tag to cakw46x2ui4e8utkudd5z0clezb0h2i0 2023-12-03T11:34:42.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.311+0000 7fcdc0321800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=cakw46x2ui4e8utkudd5z0clezb0h2i0 2023-12-03T11:34:42.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.312+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.313+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 olh_log_entry: epoch=3 op=2 key=foo[] 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 olh_log_entry: epoch=3 op=3 key=foo[EM0.ypjpco0A03015dk1mmp7pfnDcmx] 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-multipart-index-versioning.1:_:EM0.ypjpco0A03015dk1mmp7pfnDcmx_foo state=0x56553f220d98 s->prefetch_data=0 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 10 manifest: total_size = 74198 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5532246340294514608 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-multipart-index-versioning.1:_:EM0.ypjpco0A03015dk1mmp7pfnDcmx_foo state=0x56553f220d98 s->prefetch_data=0 2023-12-03T11:34:42.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f220d98 2023-12-03T11:34:42.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.314+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=20613 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=20613 stripe_ofs=20613 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4210821 part_ofs=20613 rule->part_size=20613 2023-12-03T11:34:42.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=41226 stripe_ofs=41226 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=20613 rules.size()=2 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4231434 part_ofs=41226 rule->part_size=20613 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=61839 stripe_ofs=61839 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=12359 rules.size()=2 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4252047 part_ofs=61839 rule->part_size=12359 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=74198 stripe_ofs=74198 part_ofs=74198 rule->part_size=12359 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5532246340294514608 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.1 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 344 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.2 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 588 2023-12-03T11:34:42.319 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.3 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 832 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.4 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1076 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.317+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5532246340294514608 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.318+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo[EM0.ypjpco0A03015dk1mmp7pfnDcmx] 2023-12-03T11:34:42.320 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.319+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-multipart-index-versioning.1:foo state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.322 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.321+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.322+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9.0 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.322+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.322+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.323+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.323+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.323+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.323+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.324 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.323+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-index-versioning.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.324+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 : miss 2023-12-03T11:34:42.325 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.324+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=235 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 info.flags=0x11 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 to cache LRU end 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-index-versioning.1 from cache 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-index-versioning.1, ofs0, ns] 2023-12-03T11:34:42.326 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.325+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725954 cookie 94924130870832 notifier 6039 bl.length()=193 2023-12-03T11:34:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.327+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 from cache 2023-12-03T11:34:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.327+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-index-versioning.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9, ofs0, ns] 2023-12-03T11:34:42.328 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.327+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725904 cookie 94924130845680 notifier 6039 bl.length()=250 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.329+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 : miss 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.329+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 info.flags=0x0 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 to cache LRU end 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9 from cache 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-index-versioning.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9, ofs0, ns] 2023-12-03T11:34:42.331 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.330+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725918 cookie 94924130843744 notifier 6039 bl.length()=247 2023-12-03T11:34:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 : miss 2023-12-03T11:34:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.334 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=399 2023-12-03T11:34:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 info.flags=0x17 2023-12-03T11:34:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 to cache LRU end 2023-12-03T11:34:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.335 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.333+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213990 obj=test-client.0-r-test-multipart-simple.1:foo state=0x56553f210078 s->prefetch_data=0 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 10 manifest: total_size = 886297 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10562433295627422172 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213990 obj=test-client.0-r-test-multipart-simple.1:foo state=0x56553f210078 s->prefetch_data=0 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210078 2023-12-03T11:34:42.336 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.334+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8.0 2023-12-03T11:34:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2023-12-03T11:34:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.338 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=246247 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=246247 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=738741 stripe_ofs=738741 part_ofs=738741 rule->part_size=147556 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=147556 rules.size()=2 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4928949 part_ofs=738741 rule->part_size=147556 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=886297 stripe_ofs=886297 part_ofs=886297 rule->part_size=147556 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10562433295627422172 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.1 2023-12-03T11:34:42.339 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 345 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.2 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 589 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.3 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 833 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.4 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1077 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.340 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.337+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.18tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10562433295627422172 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.338+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.338+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.338+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.339+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.339+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.339+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.339+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.341 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.339+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-multipart-simple.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.340+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 : miss 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.340+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=225 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 info.flags=0x11 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 to cache LRU end 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-multipart-simple.1 from cache 2023-12-03T11:34:42.342 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-multipart-simple.1, ofs0, ns] 2023-12-03T11:34:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.341+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725919 cookie 94924130843744 notifier 6039 bl.length()=183 2023-12-03T11:34:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.342+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 from cache 2023-12-03T11:34:42.343 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.342+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-multipart-simple.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8, ofs0, ns] 2023-12-03T11:34:42.344 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.343+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725904 cookie 94924130884384 notifier 6039 bl.length()=240 2023-12-03T11:34:42.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 : miss 2023-12-03T11:34:42.346 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 info.flags=0x0 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 to cache LRU end 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-multipart-simple.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8 from cache 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.345+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-multipart-simple.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8, ofs0, ns] 2023-12-03T11:34:42.347 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.346+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725905 cookie 94924130880688 notifier 6039 bl.length()=237 2023-12-03T11:34:42.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.348+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 : miss 2023-12-03T11:34:42.349 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.348+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=407 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 info.flags=0x17 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 to cache LRU end 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.350 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 2023-12-03T11:34:42.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:34:42.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2023-12-03T11:34:42.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2023-12-03T11:34:42.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2023-12-03T11:34:42.351 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.349+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.350+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.350+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10670431984628084925 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.350+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.350+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.352 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.350+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.353 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10670431984628084925 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_0 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 338 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 575 2023-12-03T11:34:42.354 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.352+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10670431984628084925 2023-12-03T11:34:42.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.353+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.355 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.353+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1_8288a923b35e20268a970b534527cd21 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1_8288a923b35e20268a970b534527cd21 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.356 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.354+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1_8288a923b35e20268a970b534527cd21 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_0 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 396 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 633 2023-12-03T11:34:42.358 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.356+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1_8288a923b35e20268a970b534527cd21 2023-12-03T11:34:42.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.357+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2023-12-03T11:34:42.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.359 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _fd_pJEWyfYvVj4ZZI8M9_-vveYFSElp 2023-12-03T11:34:42.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210ea8 2023-12-03T11:34:42.360 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.358+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.361 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _fd_pJEWyfYvVj4ZZI8M9_-vveYFSElp 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.t4n29GILAyGKwonzgckkVRFb6jmwJ5v_0 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.t4n29GILAyGKwonzgckkVRFb6jmwJ5v_1 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 549 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.362 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.360+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: _fd_pJEWyfYvVj4ZZI8M9_-vveYFSElp 2023-12-03T11:34:42.365 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _3nDu7fnbakQMRNXPQAOqWdyeleq8Ws6 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-FROZEN-STANDARD state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210ea8 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.362+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _3nDu7fnbakQMRNXPQAOqWdyeleq8Ws6 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.366 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.izHSb4CfBa3z3T7rXAi92UvyW5jotYr_0 2023-12-03T11:34:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:34:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.izHSb4CfBa3z3T7rXAi92UvyW5jotYr_1 2023-12-03T11:34:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:34:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.367 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.364+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _3nDu7fnbakQMRNXPQAOqWdyeleq8Ws6 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb000aa10 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4466723495057247882 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb000aa10 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210ea8 2023-12-03T11:34:42.368 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.366+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4466723495057247882 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_0 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 339 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 578 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.370 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.368+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4466723495057247882 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _UTu5_AbtjPsxnDoc4o4iVUd5OD8A350 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-FROZEN state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210ea8 2023-12-03T11:34:42.371 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.370+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.372 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.371+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _UTu5_AbtjPsxnDoc4o4iVUd5OD8A350 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.hB2mqaw_zMsmOyK_lxY373jVIBIjDxf_0 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.hB2mqaw_zMsmOyK_lxY373jVIBIjDxf_1 2023-12-03T11:34:42.374 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 545 2023-12-03T11:34:42.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.372+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: _UTu5_AbtjPsxnDoc4o4iVUd5OD8A350 2023-12-03T11:34:42.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2023-12-03T11:34:42.375 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f2122e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1_f978228b1d79971080fe4fec24e641d1 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1_f978228b1d79971080fe4fec24e641d1 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f2122e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-LUKEWARM state=0x56553f210ea8 s->prefetch_data=0 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f210ea8 2023-12-03T11:34:42.376 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.374+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1_f978228b1d79971080fe4fec24e641d1 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_0 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 398 2023-12-03T11:34:42.378 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 637 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.376+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.21tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1_f978228b1d79971080fe4fec24e641d1 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f2122e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.379 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 10 manifest: total_size = 5242880 2023-12-03T11:34:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _0xKGGxSWfhvAtKT5m_jurXGdnIkYTkp 2023-12-03T11:34:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f2122e0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-LUKEWARM-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.380 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.378+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _0xKGGxSWfhvAtKT5m_jurXGdnIkYTkp 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.yvFj7buJ7KMSfXqYQ7AkdsZMJlxOfW5_0 2023-12-03T11:34:42.382 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:34:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.yvFj7buJ7KMSfXqYQ7AkdsZMJlxOfW5_1 2023-12-03T11:34:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:34:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.381+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _0xKGGxSWfhvAtKT5m_jurXGdnIkYTkp 2023-12-03T11:34:42.383 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 10 manifest: total_size = 324647 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.398216055650256862 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.384 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.382+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=324647 stripe_ofs=324647 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.398216055650256862 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_0 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.386 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.384+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.398216055650256862 2023-12-03T11:34:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:34:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0ceaf0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 10 manifest: total_size = 324647 2023-12-03T11:34:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _YU750ByhBLheLwkrjXna69x0WwQfhQ_ 2023-12-03T11:34:42.387 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0ceaf0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-FROZEN state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.386+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=324647 stripe_ofs=324647 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _YU750ByhBLheLwkrjXna69x0WwQfhQ_ 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.qGrMPm1DHF3kob_qiprLbzjfLRUnSvs_0 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.388+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: _YU750ByhBLheLwkrjXna69x0WwQfhQ_ 2023-12-03T11:34:42.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.389+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2023-12-03T11:34:42.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 10 manifest: total_size = 324647 2023-12-03T11:34:42.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _SRv3rH0WG6BDHut34XtvK6V5qzTwCBE 2023-12-03T11:34:42.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-LUKEWARM state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.390+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.393 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=324647 stripe_ofs=324647 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _SRv3rH0WG6BDHut34XtvK6V5qzTwCBE 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.qXqEb2qIH0ywGu17MKGQc7tM95DgYfz_0 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.394 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.392+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.1tag is: _SRv3rH0WG6BDHut34XtvK6V5qzTwCBE 2023-12-03T11:34:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2023-12-03T11:34:42.395 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0ceaf0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 10 manifest: total_size = 324647 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=324647 stripe_ofs=324647 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_1_b3e931a60336b15e58c434f8cd6d3d02 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_1_b3e931a60336b15e58c434f8cd6d3d02 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0ceaf0 obj=test-client.0-r-test-obj-storage-class-copy.1:foo-STANDARD-STANDARD state=0x56553f20fb28 s->prefetch_data=0 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f20fb28 2023-12-03T11:34:42.396 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.394+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14.0 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=324647 stripe_ofs=324647 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_1_b3e931a60336b15e58c434f8cd6d3d02 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_0 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 389 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.398 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.397+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.25tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_1_b3e931a60336b15e58c434f8cd6d3d02 2023-12-03T11:34:42.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2023-12-03T11:34:42.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.399 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.400 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.398+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-copy.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 : miss 2023-12-03T11:34:42.401 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=233 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 info.flags=0x11 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 to cache LRU end 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-copy.1 from cache 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.400+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-copy.1, ofs0, ns] 2023-12-03T11:34:42.402 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.401+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725955 cookie 94924130870832 notifier 6039 bl.length()=189 2023-12-03T11:34:42.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.402+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 from cache 2023-12-03T11:34:42.403 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.402+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14, ofs0, ns] 2023-12-03T11:34:42.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.403+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725920 cookie 140521398112256 notifier 6039 bl.length()=247 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 : miss 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 info.flags=0x0 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 to cache LRU end 2023-12-03T11:34:42.407 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14 from cache 2023-12-03T11:34:42.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14, ofs0, ns] 2023-12-03T11:34:42.408 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.406+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725906 cookie 94924130845680 notifier 6039 bl.length()=244 2023-12-03T11:34:42.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 : miss 2023-12-03T11:34:42.410 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=417 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 info.flags=0x17 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 to cache LRU end 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.411 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.409+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-FROZEN 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-LUKEWARM 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN-STANDARD 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-FROZEN 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-LUKEWARM 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM-STANDARD 2023-12-03T11:34:42.412 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-FROZEN 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-LUKEWARM 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD-STANDARD 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.410+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.411+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.411+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.893260467171684828 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.411+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.411+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.413 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.411+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.893260467171684828 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 366 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 631 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 898 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2_1 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1163 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.3 2023-12-03T11:34:42.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1430 2023-12-03T11:34:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.413+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.3tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.893260467171684828 2023-12-03T11:34:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.414+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:34:42.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.414+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1_9f0c91573c7cf554525d76cb4ebcbe44 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1_9f0c91573c7cf554525d76cb4ebcbe44 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.418 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.415+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.419 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1_9f0c91573c7cf554525d76cb4ebcbe44 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 440 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 705 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 972 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2_1 2023-12-03T11:34:42.420 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1237 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.3 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1504 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.417+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.17tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1_9f0c91573c7cf554525d76cb4ebcbe44 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.418+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-FROZEN 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.418+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f21ba60 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.421 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.419+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.419+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _q48Z4DHuzOWBmcZoZnQqv7jqYjPS9gs 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.419+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f21ba60 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.419+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.419+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.422 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _q48Z4DHuzOWBmcZoZnQqv7jqYjPS9gs 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_0 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_1 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 549 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_2 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 788 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.423 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.421+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.5tag is: _q48Z4DHuzOWBmcZoZnQqv7jqYjPS9gs 2023-12-03T11:34:42.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.422+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-FROZEN-LUKEWARM 2023-12-03T11:34:42.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.422+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f21ba60 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x56553f21c8a8 s->prefetch_data=0 2023-12-03T11:34:42.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.423+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.423+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _6Q0Ha0Zene4pBzxsDkKp-XcJUm2WdLP 2023-12-03T11:34:42.424 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.423+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f21ba60 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-FROZEN-STANDARD state=0x56553f21c8a8 s->prefetch_data=0 2023-12-03T11:34:42.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.423+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21c8a8 2023-12-03T11:34:42.425 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.423+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.426 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _6Q0Ha0Zene4pBzxsDkKp-XcJUm2WdLP 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_0 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_1 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_2 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 761 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.427 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.425+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.10tag is: _6Q0Ha0Zene4pBzxsDkKp-XcJUm2WdLP 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.426+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN-STANDARD 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.426+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4009970 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x56553f21c718 s->prefetch_data=0 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.427+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.427+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17156515010278605568 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.427+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4009970 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM state=0x56553f21c718 s->prefetch_data=0 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.427+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21c718 2023-12-03T11:34:42.428 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.427+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.430 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17156515010278605568 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 374 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1 2023-12-03T11:34:42.431 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 645 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 918 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2_1 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1189 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.3 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1462 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.432 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.429+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17156515010278605568 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4047960 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _a47qrs15368aRKAANTnVoyZ_WD1PI4t 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4047960 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.433 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.430+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.434 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _a47qrs15368aRKAANTnVoyZ_WD1PI4t 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_0 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_1 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 545 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_2 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 782 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.435 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.433+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: _a47qrs15368aRKAANTnVoyZ_WD1PI4t 2023-12-03T11:34:42.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.434+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-FROZEN 2023-12-03T11:34:42.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4047960 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.436 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1_93efd03e865c66d0bee9d43ee419c831 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1_93efd03e865c66d0bee9d43ee419c831 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4047960 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.437 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.435+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.439 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1_93efd03e865c66d0bee9d43ee419c831 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 448 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 719 2023-12-03T11:34:42.440 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 992 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2_1 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1263 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.3 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1536 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.437+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.25tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1_93efd03e865c66d0bee9d43ee419c831 2023-12-03T11:34:42.441 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-LUKEWARM 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4009970 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _JK6FWFvb6xQMUICFHBJPUNRR3oPAiPR 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4009970 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-LUKEWARM-STANDARD state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.442 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.439+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _JK6FWFvb6xQMUICFHBJPUNRR3oPAiPR 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_0 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 301 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_1 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 531 2023-12-03T11:34:42.443 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_2 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 761 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.441+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.4tag is: _JK6FWFvb6xQMUICFHBJPUNRR3oPAiPR 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-LUKEWARM-STANDARD 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.444 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 10 manifest: total_size = 541681 2023-12-03T11:34:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15494616533731123648 2023-12-03T11:34:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.445 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.443+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.446 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15494616533731123648 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.1 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 365 2023-12-03T11:34:42.447 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 629 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.3 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 893 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.445+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15494616533731123648 2023-12-03T11:34:42.448 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.446+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 10 manifest: total_size = 541681 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _3KzmqdT_CDu3EYh4rRWkrzONeuINiny 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-FROZEN state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.449 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.447+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _3KzmqdT_CDu3EYh4rRWkrzONeuINiny 2023-12-03T11:34:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.450 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.4pzAWd2e03mhHYFp2VSCO_YTWobL3UC_0 2023-12-03T11:34:42.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 308 2023-12-03T11:34:42.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.449+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: _3KzmqdT_CDu3EYh4rRWkrzONeuINiny 2023-12-03T11:34:42.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.450+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.451 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.450+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-FROZEN 2023-12-03T11:34:42.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.451+0000 7fcdc0321800 10 manifest: total_size = 541681 2023-12-03T11:34:42.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.451+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to _O6nk-x8gpx308ejl5J1V6M5S-CowKAT 2023-12-03T11:34:42.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.451+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-LUKEWARM state=0x56553f21ab58 s->prefetch_data=0 2023-12-03T11:34:42.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.451+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21ab58 2023-12-03T11:34:42.452 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.451+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: _O6nk-x8gpx308ejl5J1V6M5S-CowKAT 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.31erVeHLCTZZnojAZR1pWoXkc-wXsQb_0 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 310 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.460+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.2tag is: _O6nk-x8gpx308ejl5J1V6M5S-CowKAT 2023-12-03T11:34:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-STANDARD-LUKEWARM 2023-12-03T11:34:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x56553f21da38 s->prefetch_data=0 2023-12-03T11:34:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 10 manifest: total_size = 541681 2023-12-03T11:34:42.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 10 generate_fake_tag new tag=ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2_5ec11d90dc305c0665ec50081de57f71 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2_5ec11d90dc305c0665ec50081de57f71 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-obj-storage-class-multipart-copy.1:foo-STANDARD-STANDARD state=0x56553f21da38 s->prefetch_data=0 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21da38 2023-12-03T11:34:42.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.462+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15.0 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2_5ec11d90dc305c0665ec50081de57f71 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.1 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 440 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 704 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.3 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 968 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.464+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2_5ec11d90dc305c0665ec50081de57f71 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD-STANDARD 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.466+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart-copy.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 : miss 2023-12-03T11:34:42.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=243 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 info.flags=0x11 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 to cache LRU end 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart-copy.1 from cache 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart-copy.1, ofs0, ns] 2023-12-03T11:34:42.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.468+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725906 cookie 94924130833536 notifier 6039 bl.length()=199 2023-12-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.470+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 from cache 2023-12-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.470+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart-copy.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15, ofs0, ns] 2023-12-03T11:34:42.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.470+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725907 cookie 94924130833536 notifier 6039 bl.length()=257 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 : miss 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 info.flags=0x0 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 to cache LRU end 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15 from cache 2023-12-03T11:34:42.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart-copy.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15, ofs0, ns] 2023-12-03T11:34:42.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.473+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725921 cookie 140521398112256 notifier 6039 bl.length()=254 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 : miss 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=412 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 info.flags=0x17 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 to cache LRU end 2023-12-03T11:34:42.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.476+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 2023-12-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:34:42.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f20f730 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4022595765224458958 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f20f730 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-FROZEN state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.479 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.477+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13.0 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.481 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4022595765224458958 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.1 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 367 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.1_1 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 632 2023-12-03T11:34:42.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.2 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 899 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.2_1 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1164 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.3 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1431 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.480+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.28tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4022595765224458958 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb000aa10 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 10 manifest: total_size = 11534336 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3646708246750626850 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb000aa10 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-LUKEWARM state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.484 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.481+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13.0 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=5242880 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=5242880 stripe_ofs=5242880 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=9437184 stripe_ofs=9437184 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=5242880 rules.size()=2 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=13631488 part_ofs=5242880 rule->part_size=5242880 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=1048576 rules.size()=2 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=14680064 part_ofs=10485760 rule->part_size=1048576 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=11534336 stripe_ofs=11534336 part_ofs=11534336 rule->part_size=1048576 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3646708246750626850 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.1 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 373 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.1_1 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 644 2023-12-03T11:34:42.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.2 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 917 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.2_1 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1188 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.3 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 1461 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.484+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.30tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3646708246750626850 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.485+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.485+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0d1660 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.486+0000 7fcdc0321800 10 manifest: total_size = 541681 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.486+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10190325541161139776 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.486+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0d1660 obj=test-client.0-r-test-obj-storage-class-multipart.1:foo-STANDARD state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.486+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.486+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13.0 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4190208 part_ofs=0 rule->part_size=246247 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=246247 stripe_ofs=246247 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=246247 rules.size()=2 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4436455 part_ofs=246247 rule->part_size=246247 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=492494 stripe_ofs=492494 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=49187 rules.size()=2 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): stripe_ofs=4682702 part_ofs=492494 rule->part_size=49187 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=541681 stripe_ofs=541681 part_ofs=541681 rule->part_size=49187 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10190325541161139776 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.1 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 365 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.2 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 629 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.3 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 893 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.488+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10190325541161139776 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.489+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.489+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.489+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.490+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.490+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.490+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.490+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.490+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class-multipart.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 : miss 2023-12-03T11:34:42.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=238 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 info.flags=0x11 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 to cache LRU end 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class-multipart.1 from cache 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.491+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class-multipart.1, ofs0, ns] 2023-12-03T11:34:42.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.492+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725907 cookie 94924130845680 notifier 6039 bl.length()=194 2023-12-03T11:34:42.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.493+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 from cache 2023-12-03T11:34:42.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.493+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class-multipart.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13, ofs0, ns] 2023-12-03T11:34:42.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.494+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725908 cookie 94924130833536 notifier 6039 bl.length()=252 2023-12-03T11:34:42.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 : miss 2023-12-03T11:34:42.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 info.flags=0x0 2023-12-03T11:34:42.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 to cache LRU end 2023-12-03T11:34:42.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13 from cache 2023-12-03T11:34:42.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.496+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.0 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class-multipart.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13, ofs0, ns] 2023-12-03T11:34:42.498 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.497+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725922 cookie 140521398112256 notifier 6039 bl.length()=249 2023-12-03T11:34:42.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.499+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 : miss 2023-12-03T11:34:42.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.499+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=402 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 info.flags=0x17 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 to cache LRU end 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-FROZEN 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-LUKEWARM 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 cls_bucket_list_unordered: got foo-STANDARD 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f218e20 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 10 manifest: total_size = 10485760 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11787476041521528810 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f218e20 obj=test-client.0-r-test-obj-storage-class.1:foo-FROZEN state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.500+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12.0 2023-12-03T11:34:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11787476041521528810 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_0 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 338 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_1 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 575 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_2 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 812 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.503+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.16tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11787476041521528810 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.504+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-FROZEN 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.504+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.505+0000 7fcdc0321800 10 manifest: total_size = 10485760 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.505+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14005564630534972286 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.505+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac094130 obj=test-client.0-r-test-obj-storage-class.1:foo-LUKEWARM state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.505+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.505+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12.0 2023-12-03T11:34:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14005564630534972286 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_0 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 340 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_1 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 579 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_2 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 818 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.507+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.29tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14005564630534972286 2023-12-03T11:34:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.508+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=foo-LUKEWARM 2023-12-03T11:34:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4055510 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 10 manifest: total_size = 10485760 2023-12-03T11:34:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5158658116195853606 2023-12-03T11:34:42.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4055510 obj=test-client.0-r-test-obj-storage-class.1:foo-STANDARD state=0x56553f218228 s->prefetch_data=0 2023-12-03T11:34:42.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f218228 2023-12-03T11:34:42.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.509+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12.0 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4190208 stripe_ofs=4190208 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=8380416 stripe_ofs=8380416 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=10485760 stripe_ofs=10485760 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5158658116195853606 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_0 2023-12-03T11:34:42.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 330 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_1 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 560 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_2 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 790 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.512+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.27tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5158658116195853606 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=foo-STANDARD 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.513+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-obj-storage-class.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 : miss 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=228 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 info.flags=0x11 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 to cache LRU end 2023-12-03T11:34:42.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-obj-storage-class.1 from cache 2023-12-03T11:34:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-obj-storage-class.1, ofs0, ns] 2023-12-03T11:34:42.517 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.515+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725956 cookie 94924130870832 notifier 6039 bl.length()=184 2023-12-03T11:34:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.517+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 from cache 2023-12-03T11:34:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.517+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-obj-storage-class.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12, ofs0, ns] 2023-12-03T11:34:42.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.517+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725957 cookie 94924130870832 notifier 6039 bl.length()=242 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.519+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 : miss 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 info.flags=0x0 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 to cache LRU end 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-obj-storage-class.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12 from cache 2023-12-03T11:34:42.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.7 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-obj-storage-class.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12, ofs0, ns] 2023-12-03T11:34:42.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.520+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725905 cookie 94924130884384 notifier 6039 bl.length()=239 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 : miss 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 info.flags=0x17 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 to cache LRU end 2023-12-03T11:34:42.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.523+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:34:42.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-small-obj-data.1:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 10 manifest: total_size = 0 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3160915033103728105 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-small-obj-data.1:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.524+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5.0 2023-12-03T11:34:42.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.526+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.527+0000 7fcdc0321800 10 manifest: total_size = 0 2023-12-03T11:34:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.527+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14774184527891965582 2023-12-03T11:34:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.527+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f208780 obj=test-client.0-r-test-small-obj-data.1:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.527+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.527+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5.0 2023-12-03T11:34:42.529 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.528+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:34:42.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.529+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-small-obj-data.1:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.530+0000 7fcdc0321800 10 manifest: total_size = 0 2023-12-03T11:34:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.530+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18399873132549853849 2023-12-03T11:34:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.530+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071790 obj=test-client.0-r-test-small-obj-data.1:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.530+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.530+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5.0 2023-12-03T11:34:42.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.531+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:34:42.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.532+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-small-obj-data.1:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.533+0000 7fcdc0321800 10 manifest: total_size = 0 2023-12-03T11:34:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.533+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13691691036701319401 2023-12-03T11:34:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.533+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb0071f80 obj=test-client.0-r-test-small-obj-data.1:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.533+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.533+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5.0 2023-12-03T11:34:42.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.534+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:34:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.535+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.535+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.536+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:34:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.537+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.537+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.537+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.537+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.537+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.1[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 : miss 2023-12-03T11:34:42.539 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 info.flags=0x11 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 to cache LRU end 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.1 from cache 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.538+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.1, ofs0, ns] 2023-12-03T11:34:42.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.539+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725957 cookie 94924128386896 notifier 6039 bl.length()=181 2023-12-03T11:34:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.540+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 from cache 2023-12-03T11:34:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.540+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.1:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5, ofs0, ns] 2023-12-03T11:34:42.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.541+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725923 cookie 94924130843744 notifier 6039 bl.length()=238 2023-12-03T11:34:42.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 : miss 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 info.flags=0x0 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 to cache LRU end 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5 from cache 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.543+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.1/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.5, ofs0, ns] 2023-12-03T11:34:42.545 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.544+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725909 cookie 94924130833536 notifier 6039 bl.length()=235 2023-12-03T11:34:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.546+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 : miss 2023-12-03T11:34:42.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.546+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 info.flags=0x17 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 to cache LRU end 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.548 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-small-obj-data.2:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 10 manifest: total_size = 524288 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16996394077932431862 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-small-obj-data.2:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.547+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6.0 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=524288 stripe_ofs=524288 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16996394077932431862 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.GG6UUn4-wsGBEEcK1x88pdCpCwCvqIB_0 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.550+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.6tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16996394077932431862 2023-12-03T11:34:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.551+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:34:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.551+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.552+0000 7fcdc0321800 10 manifest: total_size = 24595 2023-12-03T11:34:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.552+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9004487932909448699 2023-12-03T11:34:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.552+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f226820 obj=test-client.0-r-test-small-obj-data.2:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.552+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.553 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.552+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6.0 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=24595 stripe_ofs=24595 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9004487932909448699 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.ydjS7M4OLoX-fecWrekYvIartBD3wpt_0 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.554+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.0tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9004487932909448699 2023-12-03T11:34:42.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.555+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:34:42.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.555+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4055630 obj=test-client.0-r-test-small-obj-data.2:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.556+0000 7fcdc0321800 10 manifest: total_size = 524288 2023-12-03T11:34:42.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.556+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7325489356933763225 2023-12-03T11:34:42.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.556+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdb4055630 obj=test-client.0-r-test-small-obj-data.2:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.556+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.557 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.556+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6.0 2023-12-03T11:34:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=524288 stripe_ofs=524288 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7325489356933763225 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.5zF0ArcA0bXqB0RmQoYHXxDA_DZqbib_0 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.558+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.8tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7325489356933763225 2023-12-03T11:34:42.560 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.559+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0d1660 obj=test-client.0-r-test-small-obj-data.2:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 10 manifest: total_size = 527 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2043411954430940023 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 20 get_obj_state: octx=0x7fcdac0d1660 obj=test-client.0-r-test-small-obj-data.2:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.560+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6.0 2023-12-03T11:34:42.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=527 stripe_ofs=527 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2043411954430940023 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.rzrdPuK7suFB6IAe8G1aL20Bs9SkeEI_0 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.562+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2043411954430940023 2023-12-03T11:34:42.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:34:42.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.564+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.2[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 : miss 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 info.flags=0x11 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 to cache LRU end 2023-12-03T11:34:42.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.2 from cache 2023-12-03T11:34:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.566+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.2, ofs0, ns] 2023-12-03T11:34:42.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.567+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725958 cookie 94924128386896 notifier 6039 bl.length()=181 2023-12-03T11:34:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.568+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 from cache 2023-12-03T11:34:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.568+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.2:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6, ofs0, ns] 2023-12-03T11:34:42.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.568+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725924 cookie 94924130843744 notifier 6039 bl.length()=238 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 : miss 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 info.flags=0x0 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 to cache LRU end 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.2/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6 from cache 2023-12-03T11:34:42.572 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.2/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6, ofs0, ns] 2023-12-03T11:34:42.573 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.571+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725910 cookie 94924130833536 notifier 6039 bl.length()=235 2023-12-03T11:34:42.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.574+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 : miss 2023-12-03T11:34:42.575 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.574+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.574+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=397 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 info.flags=0x17 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 adding default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 to cache LRU end 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 updating xattr: name=user.rgw.acl bl.length()=209 2023-12-03T11:34:42.576 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 cls_bucket_list_unordered: got __ _ 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 cls_bucket_list_unordered: got ___ 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 cls_bucket_list_unordered: got obj 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 10 manifest: total_size = 49187 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3893165672394165790 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:__ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.577 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.575+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7.0 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=49187 stripe_ofs=49187 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3893165672394165790 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.2X2k7HOq0Z6tXdzOostShE79p09ga65_0 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.579 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.578+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.26tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3893165672394165790 2023-12-03T11:34:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.580 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=__ 2023-12-03T11:34:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 10 manifest: total_size = 1048576 2023-12-03T11:34:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13851552007675172905 2023-12-03T11:34:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:__ _ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.581 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.579+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7.0 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=1048576 stripe_ofs=1048576 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13851552007675172905 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.LjPJUtcZix2h9Qif6rQ6Q6P3p2NNwoQ_0 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.583 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.582+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.11tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13851552007675172905 2023-12-03T11:34:42.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.583+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=__ _ 2023-12-03T11:34:42.584 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.583+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.584+0000 7fcdc0321800 10 manifest: total_size = 1048576 2023-12-03T11:34:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.584+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5891688534375813733 2023-12-03T11:34:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.584+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:___ state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.584+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.585 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.584+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7.0 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=1048576 stripe_ofs=1048576 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5891688534375813733 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.XZPLC2lihL9d0LrHmbKE-PuIJgnPjEj_0 2023-12-03T11:34:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 328 2023-12-03T11:34:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.586+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.12tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5891688534375813733 2023-12-03T11:34:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.587+0000 7fcda27fc640 20 handle_completion(): completion ok for obj=___ 2023-12-03T11:34:42.588 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.587+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.588+0000 7fcdc0321800 10 manifest: total_size = 4134 2023-12-03T11:34:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.588+0000 7fcdc0321800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14846886487124882191 2023-12-03T11:34:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.588+0000 7fcdc0321800 20 get_obj_state: octx=0x56553f213480 obj=test-client.0-r-test-small-obj-data.3:obj state=0x56553f21d9d8 s->prefetch_data=0 2023-12-03T11:34:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.588+0000 7fcdc0321800 20 prepare_atomic_modification: state is not atomic. state=0x56553f21d9d8 2023-12-03T11:34:42.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.588+0000 7fcdc0321800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7.0 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 RGWObjManifest::operator++(): result: ofs=4134 stripe_ofs=4134 part_ofs=0 rule->part_size=0 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14846886487124882191 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - rgw_max_chunk_size is: 4194304 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - adding obj with name: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.SEMWcHTVH9c-gmA3yYa9D8UgTw_SE3q_0 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - total_encoded_size is: 329 2023-12-03T11:34:42.591 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_split_chain - sending leftover objects 2023-12-03T11:34:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.590+0000 7fcdc0321800 20 garbage collection: RGWGC::send_chain - on object name: gc.24tag is: ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14846886487124882191 2023-12-03T11:34:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.591+0000 7fcdc0321800 20 list_objects_ordered: starting attempt 1 2023-12-03T11:34:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.591+0000 7fcdc0321800 10 cls_bucket_list_ordered: request from each of 1 shard(s) for 1001 entries to get 1001 total entries 2023-12-03T11:34:42.592 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.591+0000 7fcda2ffd640 20 handle_completion(): completion ok for obj=obj 2023-12-03T11:34:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.592+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, count=0, is_truncated=0 2023-12-03T11:34:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.592+0000 7fcdc0321800 20 cls_bucket_list_ordered: returning, last_entry NOT SET 2023-12-03T11:34:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.592+0000 7fcdc0321800 10 list_objects_ordered: end of outer loop, truncated=0, count=0, attempt=1 2023-12-03T11:34:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.592+0000 7fcdc0321800 10 list_objects_ordered: exiting attempt loop because we reached end (0) or we're returning half the requested entries (0 of 1000) 2023-12-03T11:34:42.593 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.592+0000 7fcdc0321800 20 abort_multiparts INFO: aborting and cleaning up multipart upload(s); bucket=":test-client.0-r-test-small-obj-data.3[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7])"; uploads.size()=0; is_truncated=0 2023-12-03T11:34:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 10 cache get: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 : miss 2023-12-03T11:34:42.594 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 20 rados_obj.operate() r=0 bl.length=223 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 10 cache put: name=default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 info.flags=0x11 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 10 adding default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 to cache LRU end 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 10 removing default.rgw.meta+root+test-client.0-r-test-small-obj-data.3 from cache 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.593+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 1, obj: default.rgw.meta:root:test-client.0-r-test-small-obj-data.3, ofs0, ns] 2023-12-03T11:34:42.595 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.594+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725959 cookie 94924128386896 notifier 6039 bl.length()=181 2023-12-03T11:34:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.595+0000 7fcdc0321800 10 removing default.rgw.meta+root+.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 from cache 2023-12-03T11:34:42.596 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.595+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:root:.bucket.meta.test-client.0-r-test-small-obj-data.3:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7, ofs0, ns] 2023-12-03T11:34:42.597 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.596+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725925 cookie 94924130843744 notifier 6039 bl.length()=238 2023-12-03T11:34:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 10 cache get: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 : miss 2023-12-03T11:34:42.599 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 10 cache put: name=default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 info.flags=0x0 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 10 adding default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 to cache LRU end 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 10 removing default.rgw.log++pubsub..bucket.test-client.0-r-test-small-obj-data.3/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7 from cache 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.598+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.log:pubsub..bucket.test-client.0-r-test-small-obj-data.3/ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7, ofs0, ns] 2023-12-03T11:34:42.600 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.599+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725911 cookie 94924130833536 notifier 6039 bl.length()=235 2023-12-03T11:34:42.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.601+0000 7fcdc0321800 10 removing key index: GQAYCLCOVNHYCDNTYBWK 2023-12-03T11:34:42.602 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.601+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.keys:GQAYCLCOVNHYCDNTYBWK, ofs0, ns] 2023-12-03T11:34:42.603 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.601+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725926 cookie 94924130843744 notifier 6039 bl.length()=170 2023-12-03T11:34:42.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.603+0000 7fcdc0321800 10 removing email index: ragweed.client.0+test@test.test 2023-12-03T11:34:42.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.603+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.email:ragweed.client.0+test@test.test, ofs0, ns] 2023-12-03T11:34:42.604 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.603+0000 7fcda27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725958 cookie 94924130870832 notifier 6039 bl.length()=182 2023-12-03T11:34:42.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.605+0000 7fcdc0321800 10 removing user buckets index 2023-12-03T11:34:42.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.605+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0.buckets, ofs0, ns] 2023-12-03T11:34:42.606 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.605+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725908 cookie 94924130845680 notifier 6039 bl.length()=173 2023-12-03T11:34:42.607 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.606+0000 7fcdc0321800 10 removing user index: ragweed.client.0 2023-12-03T11:34:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.606+0000 7fcdc0321800 10 removing default.rgw.meta+users.uid+ragweed.client.0 from cache 2023-12-03T11:34:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.606+0000 7fcdc0321800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.uid:ragweed.client.0, ofs0, ns] 2023-12-03T11:34:42.608 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.607+0000 7fcda2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725927 cookie 94924130843744 notifier 6039 bl.length()=165 2023-12-03T11:34:42.610 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.609+0000 7fcd827e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:34:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.611+0000 7fcdc0321800 20 remove_watcher() i=0 2023-12-03T11:34:42.612 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.611+0000 7fcdc0321800 2 removed watcher, disabling cache 2023-12-03T11:34:42.613 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.612+0000 7fcdc0321800 20 remove_watcher() i=1 2023-12-03T11:34:42.614 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.613+0000 7fcdc0321800 20 remove_watcher() i=2 2023-12-03T11:34:42.615 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.614+0000 7fcdc0321800 20 remove_watcher() i=3 2023-12-03T11:34:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.615+0000 7fcdc0321800 20 remove_watcher() i=4 2023-12-03T11:34:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.616+0000 7fcdc0321800 20 remove_watcher() i=5 2023-12-03T11:34:42.618 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.617+0000 7fcdc0321800 20 remove_watcher() i=6 2023-12-03T11:34:42.619 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.618+0000 7fcdc0321800 20 remove_watcher() i=7 2023-12-03T11:34:42.624 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid sysuser.client.0 --purge-data 2023-12-03T11:34:42.706 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:34:42.706 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:34:42.718 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.717+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 realm 2023-12-03T11:34:42.720 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.719+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.720+0000 7fbb0c31f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:34:42.721 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.720+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.721+0000 7fbb0c31f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:34:42.722 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.721+0000 7fbb0c31f800 20 searching for the correct realm 2023-12-03T11:34:42.730 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.729+0000 7fbb07f54640 0 -- 172.21.15.12:0/1142883648 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55af9f179b80 msgr2=0x55af9f17bfd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.733 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:34:42.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.732+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:34:42.734 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 zone default found 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 4 Realm: () 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 zonegroup default 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.735 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.733+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.734+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.734+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.734+0000 7fbb0c31f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:34:42.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.734+0000 7fbb0c31f800 20 started sync module instance, tier type = 2023-12-03T11:34:42.736 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.734+0000 7fbb0c31f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:34:42.737 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.736+0000 7fbb07f54640 0 -- 172.21.15.12:0/1142883648 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x7fbae0079d80 msgr2=0x7fbae007c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=0 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=1 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=2 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=3 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=4 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=5 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.742+0000 7fbb0c31f800 20 add_watcher() i=6 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.743+0000 7fbb0c31f800 20 add_watcher() i=7 2023-12-03T11:34:42.744 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.743+0000 7fbb0c31f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:34:42.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.745+0000 7fbacdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:34:42.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.745+0000 7fbb0c31f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:34:42.746 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.745+0000 7fbb0c31f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:34:42.948 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.947+0000 7fbb07753640 0 --2- 172.21.15.12:0/3365674464 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55af9f1990c0 0x55af9f19f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:34:42.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fba44ff9640 20 reqs_thread_entry: start 2023-12-03T11:34:42.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fbb0c31f800 20 init_complete bucket index max shards: 1 2023-12-03T11:34:42.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fbb0c31f800 20 Filter name: none 2023-12-03T11:34:42.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fba3a7fc640 20 reqs_thread_entry: start 2023-12-03T11:34:42.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fbb0c31f800 10 cache get: name=default.rgw.meta+users.uid+sysuser.client.0 : miss 2023-12-03T11:34:42.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.991+0000 7fbb0c31f800 20 rados->read ofs=0 len=0 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 20 rados_obj.operate() r=0 bl.length=452 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 10 cache put: name=default.rgw.meta+users.uid+sysuser.client.0 info.flags=0x13 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 10 adding default.rgw.meta+users.uid+sysuser.client.0 to cache LRU end 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 10 removing key index: JNQIGRCXNWFWPGZXDSSI 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbb0c31f800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.keys:JNQIGRCXNWFWPGZXDSSI, ofs0, ns] 2023-12-03T11:34:42.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.992+0000 7fbaee7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725912 cookie 94212776816192 notifier 5783 bl.length()=170 2023-12-03T11:34:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.994+0000 7fbb0c31f800 10 removing email index: sysuser.client.0+test@test.test 2023-12-03T11:34:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.994+0000 7fbb0c31f800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 1, obj: default.rgw.meta:users.email:sysuser.client.0+test@test.test, ofs0, ns] 2023-12-03T11:34:42.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.994+0000 7fbaedffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725928 cookie 94212776817232 notifier 5783 bl.length()=182 2023-12-03T11:34:42.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.996+0000 7fbb0c31f800 10 removing user buckets index 2023-12-03T11:34:42.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.996+0000 7fbb0c31f800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 1, obj: default.rgw.meta:users.uid:sysuser.client.0.buckets, ofs0, ns] 2023-12-03T11:34:42.997 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.996+0000 7fbaee7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725913 cookie 94212776816192 notifier 5783 bl.length()=173 2023-12-03T11:34:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.997+0000 7fbb0c31f800 10 removing user index: sysuser.client.0 2023-12-03T11:34:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.997+0000 7fbb0c31f800 10 removing default.rgw.meta+users.uid+sysuser.client.0 from cache 2023-12-03T11:34:42.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.997+0000 7fbb0c31f800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 1, obj: default.rgw.meta:users.uid:sysuser.client.0, ofs0, ns] 2023-12-03T11:34:42.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:42.998+0000 7fbaedffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725959 cookie 94212776810688 notifier 5783 bl.length()=165 2023-12-03T11:34:43.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.000+0000 7fbacdfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:34:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.002+0000 7fbb0c31f800 20 remove_watcher() i=0 2023-12-03T11:34:43.003 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.002+0000 7fbb0c31f800 2 removed watcher, disabling cache 2023-12-03T11:34:43.004 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.003+0000 7fbb0c31f800 20 remove_watcher() i=1 2023-12-03T11:34:43.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.004+0000 7fbb0c31f800 20 remove_watcher() i=2 2023-12-03T11:34:43.006 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.005+0000 7fbb0c31f800 20 remove_watcher() i=3 2023-12-03T11:34:43.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.006+0000 7fbb0c31f800 20 remove_watcher() i=4 2023-12-03T11:34:43.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.007+0000 7fbb0c31f800 20 remove_watcher() i=5 2023-12-03T11:34:43.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.008+0000 7fbb0c31f800 20 remove_watcher() i=6 2023-12-03T11:34:43.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:34:43.009+0000 7fbb0c31f800 20 remove_watcher() i=7 2023-12-03T11:34:43.015 INFO:tasks.ragweed:Removing ragweed... 2023-12-03T11:34:43.015 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/ragweed.client.0 2023-12-03T11:34:43.121 INFO:teuthology.run_tasks:Running task workunit... 2023-12-03T11:34:43.131 INFO:tasks.workunit:Pulling workunits from ref bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:34:43.131 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-03T11:34:43.132 DEBUG:teuthology.orchestra.run.smithi012:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:34:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:34:43.147 INFO:teuthology.orchestra.run.smithi012.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-03T11:34:43.147 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:34:43.204 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-03T11:34:43.204 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-03T11:34:43.259 INFO:tasks.workunit:timeout=3h 2023-12-03T11:34:43.259 INFO:tasks.workunit:cleanup=True 2023-12-03T11:34:43.260 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bcbf278cf395873c2a482f5a7b2345faf2160319 2023-12-03T11:34:43.314 INFO:tasks.workunit.client.0.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-03T11:35:49.995 INFO:tasks.workunit.client.0.smithi012.stderr:Updating files: 96% (11682/12123) Updating files: 97% (11760/12123) Updating files: 98% (11881/12123) Updating files: 99% (12002/12123) Updating files: 100% (12123/12123) Updating files: 100% (12123/12123), done. 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:Note: switching to 'bcbf278cf395873c2a482f5a7b2345faf2160319'. 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:state without impacting any branches by switching back to a branch. 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-03T11:35:50.091 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: git switch -c 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr:Or undo this operation with: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: git switch - 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:50.092 INFO:tasks.workunit.client.0.smithi012.stderr:HEAD is now at bcbf278cf39 RGW:pubsub publish commit with etag populated 2023-12-03T11:35:50.094 DEBUG:teuthology.orchestra.run.smithi012:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-03T11:35:50.149 INFO:tasks.workunit.client.0.smithi012.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-03T11:35:50.151 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:35:50.151 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-03T11:35:50.203 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-03T11:35:50.255 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-03T11:35:50.296 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-03T11:35:50.297 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:35:50.297 INFO:tasks.workunit.client.0.smithi012.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-03T11:35:50.337 INFO:tasks.workunit.client.0.smithi012.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-03T11:35:50.340 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:35:50.340 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-03T11:35:50.396 INFO:tasks.workunit:Running workunits matching rgw/run-reshard.sh on client.0... 2023-12-03T11:35:50.397 INFO:tasks.workunit:Running workunit rgw/run-reshard.sh... 2023-12-03T11:35:50.398 DEBUG:teuthology.orchestra.run.smithi012:workunit test rgw/run-reshard.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bcbf278cf395873c2a482f5a7b2345faf2160319 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2023-12-03T11:35:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ ulimit -c 0 2023-12-03T11:35:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-reshard.sh 2023-12-03T11:35:50.455 INFO:tasks.workunit.client.0.smithi012.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:35:50.456 INFO:tasks.workunit.client.0.smithi012.stderr:+ python3 -m venv /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/activate 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ deactivate nondestructive 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ hash -r 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:35:52.907 INFO:tasks.workunit.client.0.smithi012.stderr:++ unset VIRTUAL_ENV 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' '!' nondestructive = nondestructive ']' 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ VIRTUAL_ENV=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ export VIRTUAL_ENV 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ _OLD_VIRTUAL_PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ PATH=/home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ export PATH 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n '' ']' 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -z '' ']' 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ _OLD_VIRTUAL_PS1= 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ PS1='(rgw) ' 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ export PS1 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:35:52.908 INFO:tasks.workunit.client.0.smithi012.stderr:++ hash -r 2023-12-03T11:35:52.909 INFO:tasks.workunit.client.0.smithi012.stderr:+ pip install pip --upgrade 2023-12-03T11:35:53.147 INFO:tasks.workunit.client.0.smithi012.stdout:Requirement already satisfied: pip in /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/lib/python3.9/site-packages (21.2.3) 2023-12-03T11:35:53.900 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting pip 2023-12-03T11:35:53.907 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached pip-23.3.1-py3-none-any.whl (2.1 MB) 2023-12-03T11:35:53.954 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: pip 2023-12-03T11:35:53.955 INFO:tasks.workunit.client.0.smithi012.stdout: Attempting uninstall: pip 2023-12-03T11:35:53.955 INFO:tasks.workunit.client.0.smithi012.stdout: Found existing installation: pip 21.2.3 2023-12-03T11:35:54.116 INFO:tasks.workunit.client.0.smithi012.stdout: Uninstalling pip-21.2.3: 2023-12-03T11:35:54.121 INFO:tasks.workunit.client.0.smithi012.stdout: Successfully uninstalled pip-21.2.3 2023-12-03T11:35:55.133 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed pip-23.3.1 2023-12-03T11:35:55.179 INFO:tasks.workunit.client.0.smithi012.stderr:+ pip install boto3 2023-12-03T11:35:56.000 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting boto3 2023-12-03T11:35:56.001 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached boto3-1.33.6-py3-none-any.whl.metadata (6.7 kB) 2023-12-03T11:35:56.583 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting botocore<1.34.0,>=1.33.6 (from boto3) 2023-12-03T11:35:56.584 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached botocore-1.33.6-py3-none-any.whl.metadata (6.1 kB) 2023-12-03T11:35:56.640 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting jmespath<2.0.0,>=0.7.1 (from boto3) 2023-12-03T11:35:56.641 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached jmespath-1.0.1-py3-none-any.whl (20 kB) 2023-12-03T11:35:56.702 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting s3transfer<0.9.0,>=0.8.2 (from boto3) 2023-12-03T11:35:56.703 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached s3transfer-0.8.2-py3-none-any.whl.metadata (1.8 kB) 2023-12-03T11:35:56.770 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:35:56.771 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2023-12-03T11:35:56.854 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting urllib3<1.27,>=1.25.4 (from botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:35:56.855 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached urllib3-1.26.18-py2.py3-none-any.whl.metadata (48 kB) 2023-12-03T11:35:57.076 INFO:tasks.workunit.client.0.smithi012.stdout:Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore<1.34.0,>=1.33.6->boto3) 2023-12-03T11:35:57.077 INFO:tasks.workunit.client.0.smithi012.stdout: Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) 2023-12-03T11:35:57.101 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached boto3-1.33.6-py3-none-any.whl (139 kB) 2023-12-03T11:35:57.102 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached botocore-1.33.6-py3-none-any.whl (11.8 MB) 2023-12-03T11:35:57.123 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached s3transfer-0.8.2-py3-none-any.whl (82 kB) 2023-12-03T11:35:57.124 INFO:tasks.workunit.client.0.smithi012.stdout:Using cached urllib3-1.26.18-py2.py3-none-any.whl (143 kB) 2023-12-03T11:35:57.228 INFO:tasks.workunit.client.0.smithi012.stdout:Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3 2023-12-03T11:35:58.301 INFO:tasks.workunit.client.0.smithi012.stdout:Successfully installed boto3-1.33.6 botocore-1.33.6 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.8.2 six-1.16.0 urllib3-1.26.18 2023-12-03T11:35:58.426 INFO:tasks.workunit.client.0.smithi012.stderr:+ /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/bin/python3 /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_rgw_reshard.py 2023-12-03T11:35:58.557 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin user create --uid tester --display-name "Testing" --access-key NX5QOQKC6BH2IDN8HC7A --secret LnEsqNNqZIpkzauboDcLXLcYaWwLQ3Kop0zAnKIn 2023-12-03T11:35:58.593 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:35:58.593 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:35:58.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.607+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 20 realm 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.608+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.609+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.609+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.610+0000 7f0329c0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:58.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.610+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.611+0000 7f0329c0f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:35:58.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.611+0000 7f0329c0f800 20 searching for the correct realm 2023-12-03T11:35:58.625 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0325a8d640 0 -- 172.21.15.12:0/2715815188 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x55ab89b53000 msgr2=0x55ab89b553f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.623+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 zone default found 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 4 Realm: () 2023-12-03T11:35:58.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 zonegroup default 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 started sync module instance, tier type = 2023-12-03T11:35:58.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.624+0000 7f0329c0f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:35:58.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f032528c640 0 -- 172.21.15.12:0/2715815188 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f031c09ceb0 msgr2=0x7f031c09ad50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=0 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=1 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=2 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=3 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=4 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=5 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.634+0000 7f0329c0f800 20 add_watcher() i=6 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.635+0000 7f0329c0f800 20 add_watcher() i=7 2023-12-03T11:35:58.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.635+0000 7f0329c0f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:35:58.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.637+0000 7f02eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:58.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.638+0000 7f0329c0f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:35:58.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.638+0000 7f0329c0f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:35:58.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0329c0f800 5 note: GC not initialized 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0259ffb640 20 reqs_thread_entry: start 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0329c0f800 20 init_complete bucket index max shards: 1 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0329c0f800 20 Filter name: none 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f024b7fe640 20 reqs_thread_entry: start 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0329c0f800 10 cache get: name=default.rgw.meta+users.uid+tester : miss 2023-12-03T11:35:58.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.644+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 adding default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : miss 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x0 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 adding default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-12-03T11:35:58.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.645+0000 7f0329c0f800 10 cache get: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A : hit (negative entry) 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.646+0000 7f0329c0f800 10 cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.646+0000 7f0329c0f800 10 moving default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.646+0000 7f0329c0f800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester, ofs0, ns] 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.647+0000 7f03077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725909 cookie 94195238201024 notifier 6081 bl.length()=535 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.647+0000 7f03077fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester info.flags=0x17 2023-12-03T11:35:58.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.647+0000 7f03077fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester to cache LRU end 2023-12-03T11:35:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f0329c0f800 10 cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-12-03T11:35:58.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f0329c0f800 10 moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:35:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f0329c0f800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:users.keys:NX5QOQKC6BH2IDN8HC7A, ofs0, ns] 2023-12-03T11:35:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f03077fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725910 cookie 94195238192976 notifier 6081 bl.length()=180 2023-12-03T11:35:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f03077fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A info.flags=0x7 2023-12-03T11:35:58.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.648+0000 7f03077fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+NX5QOQKC6BH2IDN8HC7A to cache LRU end 2023-12-03T11:35:58.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.650+0000 7f02eafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:58.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.652+0000 7f0329c0f800 20 remove_watcher() i=0 2023-12-03T11:35:58.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.652+0000 7f0329c0f800 2 removed watcher, disabling cache 2023-12-03T11:35:58.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.653+0000 7f0329c0f800 20 remove_watcher() i=1 2023-12-03T11:35:58.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.654+0000 7f0329c0f800 20 remove_watcher() i=2 2023-12-03T11:35:58.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.656+0000 7f0329c0f800 20 remove_watcher() i=3 2023-12-03T11:35:58.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.657+0000 7f0329c0f800 20 remove_watcher() i=4 2023-12-03T11:35:58.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.658+0000 7f0329c0f800 20 remove_watcher() i=5 2023-12-03T11:35:58.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.659+0000 7f0329c0f800 20 remove_watcher() i=6 2023-12-03T11:35:58.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.660+0000 7f0329c0f800 20 remove_watcher() i=7 2023-12-03T11:35:58.887 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard add 2023-12-03T11:35:58.887 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:58.889 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:35:58.918 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:35:58.919 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:35:58.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.931+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 realm 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.934+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.935+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:58.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.935+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.936+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:35:58.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.936+0000 7f4bedd2c800 20 searching for the correct realm 2023-12-03T11:35:58.947 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.946+0000 7f4bea2c4640 0 -- 172.21.15.12:0/2579638295 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556fc8a61b20 msgr2=0x556fc8a63fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:58.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:35:58.948 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:35:58.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:35:58.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:35:58.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.949 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.947+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 zone default found 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 4 Realm: () 2023-12-03T11:35:58.950 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 zonegroup default 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.948+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.949+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.949+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.949+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.949+0000 7f4bedd2c800 20 started sync module instance, tier type = 2023-12-03T11:35:58.951 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.949+0000 7f4bedd2c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:35:58.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.956+0000 7f4bea2c4640 0 -- 172.21.15.12:0/2579638295 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f4be007d440 msgr2=0x7f4be007d8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=0 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=1 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=2 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=3 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=4 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=5 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.958+0000 7f4bedd2c800 20 add_watcher() i=6 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.959+0000 7f4bedd2c800 20 add_watcher() i=7 2023-12-03T11:35:58.960 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.959+0000 7f4bedd2c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:35:58.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.961+0000 7f4baf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:58.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.961+0000 7f4bedd2c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:35:58.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.961+0000 7f4bedd2c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:35:58.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.967+0000 7f4bedd2c800 5 note: GC not initialized 2023-12-03T11:35:58.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.967+0000 7f4b1e7fc640 20 reqs_thread_entry: start 2023-12-03T11:35:58.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.967+0000 7f4bedd2c800 20 init_complete bucket index max shards: 1 2023-12-03T11:35:58.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.967+0000 7f4bedd2c800 20 Filter name: none 2023-12-03T11:35:58.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.968+0000 7f4b0ffff640 20 reqs_thread_entry: start 2023-12-03T11:35:58.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.968+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.968+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.968+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.969+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.969+0000 7f4bedd2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.969+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.969+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.969+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.970+0000 7f4bedd2c800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:35:58.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.970+0000 7f4bedd2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:35:58.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.970+0000 7f4bedd2c800 20 rados->read ofs=0 len=0 2023-12-03T11:35:58.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.971+0000 7f4bedd2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:58.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.971+0000 7f4baf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:58.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.974+0000 7f4bedd2c800 20 remove_watcher() i=0 2023-12-03T11:35:58.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.974+0000 7f4bedd2c800 2 removed watcher, disabling cache 2023-12-03T11:35:58.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.975+0000 7f4bedd2c800 20 remove_watcher() i=1 2023-12-03T11:35:58.977 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.976+0000 7f4bedd2c800 20 remove_watcher() i=2 2023-12-03T11:35:58.979 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.978+0000 7f4bedd2c800 20 remove_watcher() i=3 2023-12-03T11:35:58.980 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.979+0000 7f4bedd2c800 20 remove_watcher() i=4 2023-12-03T11:35:58.981 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.980+0000 7f4bedd2c800 20 remove_watcher() i=5 2023-12-03T11:35:58.982 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.981+0000 7f4bedd2c800 20 remove_watcher() i=6 2023-12-03T11:35:58.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:58.982+0000 7f4bedd2c800 20 remove_watcher() i=7 2023-12-03T11:35:58.987 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:58.988 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:35:58.988 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: a-bucket id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 num_objects: 0 size_kb: 0 num_shards: 1 2023-12-03T11:35:58.988 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:58.989 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 2 2023-12-03T11:35:59.019 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:35:59.019 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:35:59.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.032+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 realm 2023-12-03T11:35:59.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:35:59.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.034+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.035+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.035+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.036+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:35:59.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.036+0000 7f45ab93b800 20 searching for the correct realm 2023-12-03T11:35:59.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.045+0000 7f45a7640640 0 -- 172.21.15.12:0/2874635351 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f459c008070 msgr2=0x7f459c02f970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:59.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:35:59.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:35:59.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.047+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:35:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 zone default found 2023-12-03T11:35:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 4 Realm: () 2023-12-03T11:35:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:35:59.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 zonegroup default 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.048+0000 7f45ab93b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.049+0000 7f45ab93b800 20 started sync module instance, tier type = 2023-12-03T11:35:59.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.049+0000 7f45ab93b800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:35:59.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.054+0000 7f45a7e41640 0 -- 172.21.15.12:0/2874635351 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f459c02d450 msgr2=0x7f459c02d8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=0 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=1 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=2 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=3 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=4 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=5 2023-12-03T11:35:59.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.058+0000 7f45ab93b800 20 add_watcher() i=6 2023-12-03T11:35:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.059+0000 7f45ab93b800 20 add_watcher() i=7 2023-12-03T11:35:59.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.059+0000 7f45ab93b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:35:59.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.060+0000 7f456d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.061+0000 7f45ab93b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:35:59.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.061+0000 7f45ab93b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:35:59.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f45ab93b800 5 note: GC not initialized 2023-12-03T11:35:59.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f44dbfff640 20 reqs_thread_entry: start 2023-12-03T11:35:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f45ab93b800 20 init_complete bucket index max shards: 1 2023-12-03T11:35:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f45ab93b800 20 Filter name: none 2023-12-03T11:35:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f44d9ffb640 20 reqs_thread_entry: start 2023-12-03T11:35:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f45ab93b800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:35:59.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.067+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : miss 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.068+0000 7f45ab93b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:35:59.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.071+0000 7f456d7e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.073+0000 7f45ab93b800 20 remove_watcher() i=0 2023-12-03T11:35:59.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.073+0000 7f45ab93b800 2 removed watcher, disabling cache 2023-12-03T11:35:59.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.074+0000 7f45ab93b800 20 remove_watcher() i=1 2023-12-03T11:35:59.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.075+0000 7f45ab93b800 20 remove_watcher() i=2 2023-12-03T11:35:59.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.076+0000 7f45ab93b800 20 remove_watcher() i=3 2023-12-03T11:35:59.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.078+0000 7f45ab93b800 20 remove_watcher() i=4 2023-12-03T11:35:59.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.079+0000 7f45ab93b800 20 remove_watcher() i=5 2023-12-03T11:35:59.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.080+0000 7f45ab93b800 20 remove_watcher() i=6 2023-12-03T11:35:59.081 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.080+0000 7f45ab93b800 20 remove_watcher() i=7 2023-12-03T11:35:59.088 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard list 2023-12-03T11:35:59.113 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:35:59.114 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:35:59.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.127+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 realm 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.129+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.130+0000 7f32ab62d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.130+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.131+0000 7f32ab62d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:35:59.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.131+0000 7f32ab62d800 20 searching for the correct realm 2023-12-03T11:35:59.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.140+0000 7f32a68fa640 0 -- 172.21.15.12:0/2678732544 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f329c027980 msgr2=0x7f329c01c470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:59.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:35:59.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:35:59.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:35:59.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:35:59.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.142+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 zone default found 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 4 Realm: () 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 zonegroup default 2023-12-03T11:35:59.145 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.143+0000 7f32ab62d800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.144+0000 7f32ab62d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.144+0000 7f32ab62d800 20 started sync module instance, tier type = 2023-12-03T11:35:59.146 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.144+0000 7f32ab62d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:35:59.150 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.149+0000 7f32a68fa640 0 -- 172.21.15.12:0/2678732544 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f3280079d80 msgr2=0x7f328007c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=0 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=1 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=2 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=3 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=4 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=5 2023-12-03T11:35:59.154 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=6 2023-12-03T11:35:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 20 add_watcher() i=7 2023-12-03T11:35:59.155 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.153+0000 7f32ab62d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:35:59.156 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.155+0000 7f31e7fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.155+0000 7f32ab62d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:35:59.157 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.155+0000 7f32ab62d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:35:59.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.162+0000 7f32ab62d800 5 note: GC not initialized 2023-12-03T11:35:59.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.162+0000 7f31d6ffd640 20 reqs_thread_entry: start 2023-12-03T11:35:59.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.162+0000 7f32ab62d800 20 init_complete bucket index max shards: 1 2023-12-03T11:35:59.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.162+0000 7f32ab62d800 20 Filter name: none 2023-12-03T11:35:59.163 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.162+0000 7f31d4ff9640 20 reqs_thread_entry: start 2023-12-03T11:35:59.172 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.171+0000 7f31e7fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.173+0000 7f32ab62d800 20 remove_watcher() i=0 2023-12-03T11:35:59.174 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.173+0000 7f32ab62d800 2 removed watcher, disabling cache 2023-12-03T11:35:59.175 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.174+0000 7f32ab62d800 20 remove_watcher() i=1 2023-12-03T11:35:59.176 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.175+0000 7f32ab62d800 20 remove_watcher() i=2 2023-12-03T11:35:59.177 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.176+0000 7f32ab62d800 20 remove_watcher() i=3 2023-12-03T11:35:59.178 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.177+0000 7f32ab62d800 20 remove_watcher() i=4 2023-12-03T11:35:59.179 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.178+0000 7f32ab62d800 20 remove_watcher() i=5 2023-12-03T11:35:59.180 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.179+0000 7f32ab62d800 20 remove_watcher() i=6 2023-12-03T11:35:59.181 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.180+0000 7f32ab62d800 20 remove_watcher() i=7 2023-12-03T11:35:59.186 INFO:tasks.workunit.client.0.smithi012.stderr:bucket name a-bucket 2023-12-03T11:35:59.186 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard process 2023-12-03T11:35:59.187 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:35:59.188 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard process 2023-12-03T11:35:59.213 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:35:59.213 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:35:59.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.226+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 realm 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.228+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.230+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.230+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.230+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:35:59.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.230+0000 7fe6ebd1f800 20 searching for the correct realm 2023-12-03T11:35:59.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.240+0000 7fe6e794d640 0 -- 172.21.15.12:0/3940282446 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55f2218bc600 msgr2=0x55f2218be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:35:59.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.241+0000 7fe6ebd1f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:35:59.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.241+0000 7fe6ebd1f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:35:59.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.241+0000 7fe6ebd1f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:35:59.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.241+0000 7fe6ebd1f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:35:59.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.241+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 zone default found 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 4 Realm: () 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 zonegroup default 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.242+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:35:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 started sync module instance, tier type = 2023-12-03T11:35:59.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.243+0000 7fe6ebd1f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:35:59.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.248+0000 7fe6e814e640 0 -- 172.21.15.12:0/3940282446 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55f2218c4500 msgr2=0x55f2218c6990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:35:59.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=0 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=1 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=2 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=3 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=4 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=5 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.252+0000 7fe6ebd1f800 20 add_watcher() i=6 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.253+0000 7fe6ebd1f800 20 add_watcher() i=7 2023-12-03T11:35:59.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.253+0000 7fe6ebd1f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:35:59.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.255+0000 7fe6adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.255+0000 7fe6ebd1f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:35:59.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.255+0000 7fe6ebd1f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.261+0000 7fe6ebd1f800 5 note: GC not initialized 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.261+0000 7fe624ff9640 20 reqs_thread_entry: start 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.261+0000 7fe6ebd1f800 20 init_complete bucket index max shards: 1 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.262+0000 7fe6ebd1f800 20 Filter name: none 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.262+0000 7fe61a7fc640 20 reqs_thread_entry: start 2023-12-03T11:35:59.263 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.262+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000000 2023-12-03T11:35:59.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.265+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000000 , ret = 0 2023-12-03T11:35:59.266 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.265+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000001 2023-12-03T11:35:59.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.267+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000001 , ret = 0 2023-12-03T11:35:59.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.267+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000002 2023-12-03T11:35:59.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.270+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000002 , ret = 0 2023-12-03T11:35:59.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.270+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000003 2023-12-03T11:35:59.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.272+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000003 , ret = 0 2023-12-03T11:35:59.273 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.272+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000004 2023-12-03T11:35:59.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.275+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000004 , ret = 0 2023-12-03T11:35:59.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.275+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000005 2023-12-03T11:35:59.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.277+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000005 , ret = 0 2023-12-03T11:35:59.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.277+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000006 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.279+0000 7fe6ebd1f800 20 process_entry resharding a-bucket 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.279+0000 7fe6ebd1f800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.279+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : miss 2023-12-03T11:35:59.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 20 rados->read ofs=0 len=0 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.280+0000 7fe6ebd1f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:35:59.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.285+0000 7fe6ebd1f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.285+0000 7fe6ebd1f800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.285+0000 7fe6ebd1f800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.285+0000 7fe6ebd1f800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ce7fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725914 cookie 94498433313616 notifier 5882 bl.length()=789 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ce7fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ce7fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ce7fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:35:59.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.286+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.287+0000 7fe6ebd1f800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.0 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.287+0000 7fe6ebd1f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.0 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.287+0000 7fe6ebd1f800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.1 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.287+0000 7fe6ebd1f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.1 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.287+0000 7fe6ebd1f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.0 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.288+0000 7fe6ebd1f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:35:59.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.288+0000 7fe6ebd1f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.289+0000 7fe6ebd1f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.289+0000 7fe6ebd1f800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.289+0000 7fe6ebd1f800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.289+0000 7fe6ebd1f800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6cdffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725915 cookie 94498433313616 notifier 5882 bl.length()=757 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6cdffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:35:59.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6cdffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6cdffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.290+0000 7fe6ebd1f800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:35:59.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.291+0000 7fe6ebd1f800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:35:59.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.292+0000 7fe6ebd1f800 20 process_entry removing reshard queue entry for bucket a-bucket 2023-12-03T11:35:59.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.294+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000006 , ret = 0 2023-12-03T11:35:59.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.294+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000007 2023-12-03T11:35:59.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.296+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000007 , ret = 0 2023-12-03T11:35:59.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.296+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000008 2023-12-03T11:35:59.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.298+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000008 , ret = 0 2023-12-03T11:35:59.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.298+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000009 2023-12-03T11:35:59.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.301+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000009 , ret = 0 2023-12-03T11:35:59.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.301+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000010 2023-12-03T11:35:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.303+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000010 , ret = 0 2023-12-03T11:35:59.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.303+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000011 2023-12-03T11:35:59.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.305+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000011 , ret = 0 2023-12-03T11:35:59.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.305+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000012 2023-12-03T11:35:59.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.308+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000012 , ret = 0 2023-12-03T11:35:59.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.308+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000013 2023-12-03T11:35:59.311 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.310+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000013 , ret = 0 2023-12-03T11:35:59.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.310+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000014 2023-12-03T11:35:59.313 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.312+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000014 , ret = 0 2023-12-03T11:35:59.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.312+0000 7fe6ebd1f800 20 processing logshard = reshard.0000000015 2023-12-03T11:35:59.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.315+0000 7fe6ebd1f800 20 finish processing logshard = reshard.0000000015 , ret = 0 2023-12-03T11:35:59.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.316+0000 7fe6adfe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:35:59.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.318+0000 7fe6ebd1f800 20 remove_watcher() i=0 2023-12-03T11:35:59.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.318+0000 7fe6ebd1f800 2 removed watcher, disabling cache 2023-12-03T11:35:59.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.319+0000 7fe6ebd1f800 20 remove_watcher() i=1 2023-12-03T11:35:59.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.320+0000 7fe6ebd1f800 20 remove_watcher() i=2 2023-12-03T11:35:59.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.321+0000 7fe6ebd1f800 20 remove_watcher() i=3 2023-12-03T11:35:59.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.322+0000 7fe6ebd1f800 20 remove_watcher() i=4 2023-12-03T11:35:59.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.323+0000 7fe6ebd1f800 20 remove_watcher() i=5 2023-12-03T11:35:59.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.324+0000 7fe6ebd1f800 20 remove_watcher() i=6 2023-12-03T11:35:59.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:35:59.325+0000 7fe6ebd1f800 20 remove_watcher() i=7 2023-12-03T11:36:04.337 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:36:04.371 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:04.371 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:04.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.577+0000 7fa43f94d640 0 --2- 172.21.15.12:0/2924710011 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x55c08b0e5990 0x55c08b242010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:36:04.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.588+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 20 realm 2023-12-03T11:36:04.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.590+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.591+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.591+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.592+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.592+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.594+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:04.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.594+0000 7fa443d25800 20 searching for the correct realm 2023-12-03T11:36:04.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.603+0000 7fa43f14c640 0 -- 172.21.15.12:0/1867178707 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55c08b2af060 msgr2=0x55c08b2b1450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.607+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 zone default found 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 4 Realm: () 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 zonegroup default 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.609+0000 7fa44014e640 0 -- 172.21.15.12:0/1867178707 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55c08b2b3920 msgr2=0x55c08b2b5de0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.610+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.610+0000 7fa443d25800 20 started sync module instance, tier type = 2023-12-03T11:36:04.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.610+0000 7fa443d25800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=0 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=1 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=2 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=3 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=4 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=5 2023-12-03T11:36:04.623 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.621+0000 7fa443d25800 20 add_watcher() i=6 2023-12-03T11:36:04.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.622+0000 7fa443d25800 20 add_watcher() i=7 2023-12-03T11:36:04.624 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.622+0000 7fa443d25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:04.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.624+0000 7fa405fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.624+0000 7fa443d25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:04.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.624+0000 7fa443d25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.631+0000 7fa443d25800 5 note: GC not initialized 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.631+0000 7fa37cff9640 20 reqs_thread_entry: start 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.631+0000 7fa443d25800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.632+0000 7fa443d25800 20 Filter name: none 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.632+0000 7fa3727fc640 20 reqs_thread_entry: start 2023-12-03T11:36:04.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.632+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.633+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:04.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.633+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.633+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.633+0000 7fa443d25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.633+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.634+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.634+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.634+0000 7fa443d25800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:04.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.634+0000 7fa443d25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:04.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.635+0000 7fa443d25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.635+0000 7fa443d25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.636+0000 7fa405fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.638+0000 7fa443d25800 20 remove_watcher() i=0 2023-12-03T11:36:04.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.638+0000 7fa443d25800 2 removed watcher, disabling cache 2023-12-03T11:36:04.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.641+0000 7fa443d25800 20 remove_watcher() i=1 2023-12-03T11:36:04.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.642+0000 7fa443d25800 20 remove_watcher() i=2 2023-12-03T11:36:04.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.642+0000 7fa443d25800 20 remove_watcher() i=3 2023-12-03T11:36:04.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.643+0000 7fa443d25800 20 remove_watcher() i=4 2023-12-03T11:36:04.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.644+0000 7fa443d25800 20 remove_watcher() i=5 2023-12-03T11:36:04.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.645+0000 7fa443d25800 20 remove_watcher() i=6 2023-12-03T11:36:04.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.647+0000 7fa443d25800 20 remove_watcher() i=7 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: a-bucket id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 num_objects: 0 size_kb: 0 num_shards: 2 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard add non empty bucket 2023-12-03T11:36:04.655 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:04.727 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:36:04.755 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:04.755 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:04.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.767+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 20 realm 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.769+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.770+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.770+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.770+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.770+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.771+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:04.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.771+0000 7f3b42b25800 20 searching for the correct realm 2023-12-03T11:36:04.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.781+0000 7f3b3e754640 0 -- 172.21.15.12:0/193845592 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x563fd0739c80 msgr2=0x563fd073c090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.782+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:04.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 zone default found 2023-12-03T11:36:04.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 4 Realm: () 2023-12-03T11:36:04.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 zonegroup default 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.783+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.784+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.784+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.784+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.784+0000 7f3b42b25800 20 started sync module instance, tier type = 2023-12-03T11:36:04.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.784+0000 7f3b42b25800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:04.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.790+0000 7f3b3ef55640 0 -- 172.21.15.12:0/193845592 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f3b2802b750 msgr2=0x7f3b280093c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=0 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=1 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=2 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=3 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=4 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=5 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.793+0000 7f3b42b25800 20 add_watcher() i=6 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.794+0000 7f3b42b25800 20 add_watcher() i=7 2023-12-03T11:36:04.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.794+0000 7f3b42b25800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:04.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.796+0000 7f3b03fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.796+0000 7f3b42b25800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:04.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.796+0000 7f3b42b25800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.802+0000 7f3b42b25800 5 note: GC not initialized 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.802+0000 7f3a72ffd640 20 reqs_thread_entry: start 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.802+0000 7f3b42b25800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3b42b25800 20 Filter name: none 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3a70ff9640 20 reqs_thread_entry: start 2023-12-03T11:36:04.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.803+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:04.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:04.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.804+0000 7f3b42b25800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:04.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.805+0000 7f3b42b25800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.805+0000 7f3b42b25800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.806+0000 7f3b03fe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.809+0000 7f3b42b25800 20 remove_watcher() i=0 2023-12-03T11:36:04.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.809+0000 7f3b42b25800 2 removed watcher, disabling cache 2023-12-03T11:36:04.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.810+0000 7f3b42b25800 20 remove_watcher() i=1 2023-12-03T11:36:04.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.810+0000 7f3b42b25800 20 remove_watcher() i=2 2023-12-03T11:36:04.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.811+0000 7f3b42b25800 20 remove_watcher() i=3 2023-12-03T11:36:04.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.812+0000 7f3b42b25800 20 remove_watcher() i=4 2023-12-03T11:36:04.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.813+0000 7f3b42b25800 20 remove_watcher() i=5 2023-12-03T11:36:04.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.814+0000 7f3b42b25800 20 remove_watcher() i=6 2023-12-03T11:36:04.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.815+0000 7f3b42b25800 20 remove_watcher() i=7 2023-12-03T11:36:04.822 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:04.822 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:04.822 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: a-bucket id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 num_objects: 8 size_kb: 1 num_shards: 2 2023-12-03T11:36:04.822 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:04.823 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard add --bucket a-bucket --num-shards 3 2023-12-03T11:36:04.853 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:04.853 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:04.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.865+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 realm 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:04.872 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.867+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.868+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.868+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.868+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.869+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:04.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.869+0000 7f272be8c800 20 searching for the correct realm 2023-12-03T11:36:04.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.878+0000 7f272998c640 0 -- 172.21.15.12:0/864774258 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556da8df9720 msgr2=0x556da8dfbb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:04.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:04.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:04.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:04.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.880+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 zone default found 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 4 Realm: () 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:04.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 zonegroup default 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 started sync module instance, tier type = 2023-12-03T11:36:04.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.881+0000 7f272be8c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:04.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.887+0000 7f272a18d640 0 -- 172.21.15.12:0/864774258 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f2714008410 msgr2=0x7f2714008890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=0 2023-12-03T11:36:04.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=1 2023-12-03T11:36:04.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=2 2023-12-03T11:36:04.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=3 2023-12-03T11:36:04.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=4 2023-12-03T11:36:04.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=5 2023-12-03T11:36:04.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.890+0000 7f272be8c800 20 add_watcher() i=6 2023-12-03T11:36:04.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.891+0000 7f272be8c800 20 add_watcher() i=7 2023-12-03T11:36:04.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.891+0000 7f272be8c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:04.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.892+0000 7f26ef7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.894+0000 7f272be8c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:04.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.894+0000 7f272be8c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f272be8c800 5 note: GC not initialized 2023-12-03T11:36:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f265e7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:04.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f272be8c800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:04.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f272be8c800 20 Filter name: none 2023-12-03T11:36:04.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f264ffff640 20 reqs_thread_entry: start 2023-12-03T11:36:04.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f272be8c800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:36:04.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.899+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : miss 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:04.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.900+0000 7f272be8c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:36:04.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.902+0000 7f26ef7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.904+0000 7f272be8c800 20 remove_watcher() i=0 2023-12-03T11:36:04.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.904+0000 7f272be8c800 2 removed watcher, disabling cache 2023-12-03T11:36:04.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.905+0000 7f272be8c800 20 remove_watcher() i=1 2023-12-03T11:36:04.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.906+0000 7f272be8c800 20 remove_watcher() i=2 2023-12-03T11:36:04.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.907+0000 7f272be8c800 20 remove_watcher() i=3 2023-12-03T11:36:04.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.908+0000 7f272be8c800 20 remove_watcher() i=4 2023-12-03T11:36:04.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.909+0000 7f272be8c800 20 remove_watcher() i=5 2023-12-03T11:36:04.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.910+0000 7f272be8c800 20 remove_watcher() i=6 2023-12-03T11:36:04.913 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.911+0000 7f272be8c800 20 remove_watcher() i=7 2023-12-03T11:36:04.921 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard list 2023-12-03T11:36:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:04.945 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:04.959 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.958+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 realm 2023-12-03T11:36:04.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.961 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.960+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.961+0000 7f0230532800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.962 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.961+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.961+0000 7f0230532800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:04.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.961+0000 7f0230532800 20 searching for the correct realm 2023-12-03T11:36:04.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f022c162640 0 -- 172.21.15.12:0/2574533590 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55d94303bee0 msgr2=0x55d94303e2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:04.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:04.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.972+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:04.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 zone default found 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 4 Realm: () 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 zonegroup default 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.973+0000 7f0230532800 20 rados->read ofs=0 len=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.974+0000 7f0230532800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.974+0000 7f0230532800 20 started sync module instance, tier type = 2023-12-03T11:36:04.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.974+0000 7f0230532800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:04.983 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.980+0000 7f022c162640 0 -- 172.21.15.12:0/2574533590 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f022007d3c0 msgr2=0x7f022007d840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=0 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=1 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=2 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=3 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=4 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=5 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.983+0000 7f0230532800 20 add_watcher() i=6 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.984+0000 7f0230532800 20 add_watcher() i=7 2023-12-03T11:36:04.985 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.984+0000 7f0230532800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:04.987 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.986+0000 7f01f27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:04.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.986+0000 7f0230532800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:04.988 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.986+0000 7f0230532800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:04.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.992+0000 7f0230532800 5 note: GC not initialized 2023-12-03T11:36:04.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.993+0000 7f01617fa640 20 reqs_thread_entry: start 2023-12-03T11:36:04.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.993+0000 7f0230532800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:04.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.993+0000 7f0230532800 20 Filter name: none 2023-12-03T11:36:04.994 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:04.993+0000 7f015affd640 20 reqs_thread_entry: start 2023-12-03T11:36:05.003 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.002+0000 7f01f27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.004+0000 7f0230532800 20 remove_watcher() i=0 2023-12-03T11:36:05.005 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.004+0000 7f0230532800 2 removed watcher, disabling cache 2023-12-03T11:36:05.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.005+0000 7f0230532800 20 remove_watcher() i=1 2023-12-03T11:36:05.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.006+0000 7f0230532800 20 remove_watcher() i=2 2023-12-03T11:36:05.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.007+0000 7f0230532800 20 remove_watcher() i=3 2023-12-03T11:36:05.009 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.008+0000 7f0230532800 20 remove_watcher() i=4 2023-12-03T11:36:05.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.009+0000 7f0230532800 20 remove_watcher() i=5 2023-12-03T11:36:05.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.010+0000 7f0230532800 20 remove_watcher() i=6 2023-12-03T11:36:05.012 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.011+0000 7f0230532800 20 remove_watcher() i=7 2023-12-03T11:36:05.017 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard process non empty bucket 2023-12-03T11:36:05.017 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.019 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin reshard process 2023-12-03T11:36:05.044 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:05.044 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:05.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.058+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.059+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.059+0000 7f1cfbfaf800 20 realm 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.059+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.060+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.060+0000 7f1cfbfaf800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.060+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.060+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.060+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.061+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.061+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.062+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:05.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.062+0000 7f1cfbfaf800 20 searching for the correct realm 2023-12-03T11:36:05.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.071+0000 7f1cf9aaf640 0 -- 172.21.15.12:0/3161856721 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5587d739ac20 msgr2=0x5587d739d010 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.072+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.074 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 zone default found 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 4 Realm: () 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:05.075 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 zonegroup default 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.073+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.074+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.074+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.074+0000 7f1cfbfaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.074+0000 7f1cfbfaf800 20 started sync module instance, tier type = 2023-12-03T11:36:05.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.074+0000 7f1cfbfaf800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:05.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.077+0000 7f1cf9aaf640 0 -- 172.21.15.12:0/3161856721 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5587d73a2cb0 msgr2=0x7f1cd4081730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:05.084 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.083+0000 7f1cfbfaf800 20 add_watcher() i=0 2023-12-03T11:36:05.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=1 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=2 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=3 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=4 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=5 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=6 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 20 add_watcher() i=7 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.280+0000 7f1cfbfaf800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:05.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.282+0000 7f1cbf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.282+0000 7f1cfbfaf800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:05.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.282+0000 7f1cfbfaf800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:05.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1cfbfaf800 5 note: GC not initialized 2023-12-03T11:36:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1c2e7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1cfbfaf800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1cfbfaf800 20 Filter name: none 2023-12-03T11:36:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1c1ffff640 20 reqs_thread_entry: start 2023-12-03T11:36:05.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.288+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000000 2023-12-03T11:36:05.292 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.292+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000000 , ret = 0 2023-12-03T11:36:05.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.292+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000001 2023-12-03T11:36:05.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.294+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000001 , ret = 0 2023-12-03T11:36:05.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.294+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000002 2023-12-03T11:36:05.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.297+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000002 , ret = 0 2023-12-03T11:36:05.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.297+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000003 2023-12-03T11:36:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.300+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000003 , ret = 0 2023-12-03T11:36:05.301 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.300+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000004 2023-12-03T11:36:05.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.302+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000004 , ret = 0 2023-12-03T11:36:05.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.302+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000005 2023-12-03T11:36:05.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.306+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000005 , ret = 0 2023-12-03T11:36:05.307 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.306+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000006 2023-12-03T11:36:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.307+0000 7f1cfbfaf800 20 process_entry resharding a-bucket 2023-12-03T11:36:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.307+0000 7f1cfbfaf800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:36:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.307+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:05.309 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x1 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : miss 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.310 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.308+0000 7f1cfbfaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cdbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725916 cookie 94041919877536 notifier 6267 bl.length()=789 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cdbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cdbfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cdbfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:36:05.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:36:05.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:36:05.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.314+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.0 2023-12-03T11:36:05.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.0 2023-12-03T11:36:05.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.1 2023-12-03T11:36:05.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.316+0000 7f1cfbfaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.1.1 2023-12-03T11:36:05.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.318+0000 7f1cfbfaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:05.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.318+0000 7f1cfbfaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:36:05.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cfbfaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cfbfaf800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cfbfaf800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cfbfaf800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:36:05.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cdbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725917 cookie 94041919877536 notifier 6267 bl.length()=757 2023-12-03T11:36:05.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cdbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cdbfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.319+0000 7f1cdbfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cdbfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725918 cookie 94041919877536 notifier 6267 bl.length()=757 2023-12-03T11:36:05.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cdbfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cdbfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cdbfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:36:05.323 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.321+0000 7f1cfbfaf800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:36:05.324 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.323+0000 7f1cfbfaf800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:36:05.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.324+0000 7f1cfbfaf800 20 process_entry removing reshard queue entry for bucket a-bucket 2023-12-03T11:36:05.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.326+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000006 , ret = 0 2023-12-03T11:36:05.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.326+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000007 2023-12-03T11:36:05.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.328+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000007 , ret = 0 2023-12-03T11:36:05.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.328+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000008 2023-12-03T11:36:05.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.330+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000008 , ret = 0 2023-12-03T11:36:05.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.330+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000009 2023-12-03T11:36:05.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.332+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000009 , ret = 0 2023-12-03T11:36:05.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.332+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000010 2023-12-03T11:36:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.335+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000010 , ret = 0 2023-12-03T11:36:05.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.335+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000011 2023-12-03T11:36:05.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.338+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000011 , ret = 0 2023-12-03T11:36:05.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.338+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000012 2023-12-03T11:36:05.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.340+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000012 , ret = 0 2023-12-03T11:36:05.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.340+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000013 2023-12-03T11:36:05.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.343+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000013 , ret = 0 2023-12-03T11:36:05.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.343+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000014 2023-12-03T11:36:05.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.345+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000014 , ret = 0 2023-12-03T11:36:05.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.345+0000 7f1cfbfaf800 20 processing logshard = reshard.0000000015 2023-12-03T11:36:05.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.348+0000 7f1cfbfaf800 20 finish processing logshard = reshard.0000000015 , ret = 0 2023-12-03T11:36:05.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.348+0000 7f1cbf7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.351+0000 7f1cfbfaf800 20 remove_watcher() i=0 2023-12-03T11:36:05.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.351+0000 7f1cfbfaf800 2 removed watcher, disabling cache 2023-12-03T11:36:05.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.352+0000 7f1cfbfaf800 20 remove_watcher() i=1 2023-12-03T11:36:05.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.353+0000 7f1cfbfaf800 20 remove_watcher() i=2 2023-12-03T11:36:05.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.354+0000 7f1cfbfaf800 20 remove_watcher() i=3 2023-12-03T11:36:05.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.355+0000 7f1cfbfaf800 20 remove_watcher() i=4 2023-12-03T11:36:05.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.356+0000 7f1cfbfaf800 20 remove_watcher() i=5 2023-12-03T11:36:05.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.357+0000 7f1cfbfaf800 20 remove_watcher() i=6 2023-12-03T11:36:05.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.358+0000 7f1cfbfaf800 20 remove_watcher() i=7 2023-12-03T11:36:05.365 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:36:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:05.393 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:05.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.406+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 realm 2023-12-03T11:36:05.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.409 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.408+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.409+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.409+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.410+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:05.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.410+0000 7f799e330800 20 searching for the correct realm 2023-12-03T11:36:05.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.419+0000 7f799a0c6640 0 -- 172.21.15.12:0/2560788512 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x558fd09204b0 msgr2=0x558fd09228a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.420+0000 7f799e330800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.420+0000 7f799e330800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.420+0000 7f799e330800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.420+0000 7f799e330800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.420+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.423 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 zone default found 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 4 Realm: () 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 zonegroup default 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.421+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 started sync module instance, tier type = 2023-12-03T11:36:05.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.422+0000 7f799e330800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:05.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.429+0000 7f799a8c7640 0 -- 172.21.15.12:0/2560788512 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x558fd09281b0 msgr2=0x558fd092a650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.432 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=0 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=1 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=2 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=3 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=4 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=5 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.431+0000 7f799e330800 20 add_watcher() i=6 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.432+0000 7f799e330800 20 add_watcher() i=7 2023-12-03T11:36:05.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.432+0000 7f799e330800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:05.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.434+0000 7f795ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.434+0000 7f799e330800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:05.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.434+0000 7f799e330800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.440+0000 7f799e330800 5 note: GC not initialized 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f78ceffd640 20 reqs_thread_entry: start 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f799e330800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f799e330800 20 Filter name: none 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f78ccff9640 20 reqs_thread_entry: start 2023-12-03T11:36:05.442 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.441+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.442+0000 7f799e330800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:36:05.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.443+0000 7f799e330800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.443+0000 7f799e330800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.443+0000 7f799e330800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.444+0000 7f795ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.446+0000 7f799e330800 20 remove_watcher() i=0 2023-12-03T11:36:05.447 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.446+0000 7f799e330800 2 removed watcher, disabling cache 2023-12-03T11:36:05.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.447+0000 7f799e330800 20 remove_watcher() i=1 2023-12-03T11:36:05.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.448+0000 7f799e330800 20 remove_watcher() i=2 2023-12-03T11:36:05.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.449+0000 7f799e330800 20 remove_watcher() i=3 2023-12-03T11:36:05.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.450+0000 7f799e330800 20 remove_watcher() i=4 2023-12-03T11:36:05.452 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.451+0000 7f799e330800 20 remove_watcher() i=5 2023-12-03T11:36:05.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.452+0000 7f799e330800 20 remove_watcher() i=6 2023-12-03T11:36:05.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.453+0000 7f799e330800 20 remove_watcher() i=7 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: a-bucket id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 num_objects: 8 size_kb: 1 num_shards: 3 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with EIO injected at set_target_layout 2023-12-03T11:36:05.459 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.687 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:36:05.714 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:05.714 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:05.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.728+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.729+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.729+0000 7f7e15b2c800 20 realm 2023-12-03T11:36:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.729+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.729+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.729+0000 7f7e15b2c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:05.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.730+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.730+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.730+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.731+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.731+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.731+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:05.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.731+0000 7f7e15b2c800 20 searching for the correct realm 2023-12-03T11:36:05.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.741+0000 7f7e118f0640 0 -- 172.21.15.12:0/3915589599 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f7e0800abc0 msgr2=0x7f7e08025920 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.742+0000 7f7e15b2c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.742+0000 7f7e15b2c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.742+0000 7f7e15b2c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.742+0000 7f7e15b2c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.742+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 zone default found 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 4 Realm: () 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 zonegroup default 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.743+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 started sync module instance, tier type = 2023-12-03T11:36:05.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.744+0000 7f7e15b2c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:05.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.749+0000 7f7e120f1640 0 -- 172.21.15.12:0/3915589599 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f7e08024400 msgr2=0x7f7e08024860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=0 2023-12-03T11:36:05.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=1 2023-12-03T11:36:05.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=2 2023-12-03T11:36:05.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=3 2023-12-03T11:36:05.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=4 2023-12-03T11:36:05.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=5 2023-12-03T11:36:05.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.753+0000 7f7e15b2c800 20 add_watcher() i=6 2023-12-03T11:36:05.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.754+0000 7f7e15b2c800 20 add_watcher() i=7 2023-12-03T11:36:05.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.754+0000 7f7e15b2c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:05.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.755+0000 7f7dd77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.756+0000 7f7e15b2c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:05.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.756+0000 7f7e15b2c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:05.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7e15b2c800 5 note: GC not initialized 2023-12-03T11:36:05.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7d467fc640 20 reqs_thread_entry: start 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7e15b2c800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7e15b2c800 20 Filter name: none 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7d37fff640 20 reqs_thread_entry: start 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.762+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.763+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:05.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.763+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.764+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:05.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.764+0000 7f7e15b2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.764+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.764+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:05.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.764+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.765+0000 7f7e15b2c800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:05.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.765+0000 7f7e15b2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.765+0000 7f7e15b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.765+0000 7f7e15b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.766+0000 7f7dd77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.769+0000 7f7e15b2c800 20 remove_watcher() i=0 2023-12-03T11:36:05.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.769+0000 7f7e15b2c800 2 removed watcher, disabling cache 2023-12-03T11:36:05.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.770+0000 7f7e15b2c800 20 remove_watcher() i=1 2023-12-03T11:36:05.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.771+0000 7f7e15b2c800 20 remove_watcher() i=2 2023-12-03T11:36:05.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.772+0000 7f7e15b2c800 20 remove_watcher() i=3 2023-12-03T11:36:05.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.773+0000 7f7e15b2c800 20 remove_watcher() i=4 2023-12-03T11:36:05.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.774+0000 7f7e15b2c800 20 remove_watcher() i=5 2023-12-03T11:36:05.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.775+0000 7f7e15b2c800 20 remove_watcher() i=6 2023-12-03T11:36:05.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.776+0000 7f7e15b2c800 20 remove_watcher() i=7 2023-12-03T11:36:05.782 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.782 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:05.782 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:05.782 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.784 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout 2023-12-03T11:36:05.817 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:05.817 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:05.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.830+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 realm 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.832+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.833+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.833+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.834+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:05.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.834+0000 7f5b29b27800 20 searching for the correct realm 2023-12-03T11:36:05.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.842+0000 7f5b258bf640 0 -- 172.21.15.12:0/109396642 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f5b1c072920 msgr2=0x7f5b1c08a3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 zone default found 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 4 Realm: () 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 zonegroup default 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.844+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 started sync module instance, tier type = 2023-12-03T11:36:05.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.845+0000 7f5b29b27800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:05.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.850+0000 7f5b250be640 0 -- 172.21.15.12:0/109396642 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x558942560170 msgr2=0x558942562560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=0 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=1 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=2 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=3 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=4 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=5 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.854+0000 7f5b29b27800 20 add_watcher() i=6 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.855+0000 7f5b29b27800 20 add_watcher() i=7 2023-12-03T11:36:05.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.855+0000 7f5b29b27800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:05.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.857+0000 7f5aeb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.857+0000 7f5b29b27800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:05.858 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.857+0000 7f5b29b27800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.863+0000 7f5b29b27800 5 note: GC not initialized 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.863+0000 7f5a5a7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.863+0000 7f5b29b27800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.863+0000 7f5b29b27800 20 Filter name: none 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.863+0000 7f5a4bfff640 20 reqs_thread_entry: start 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:36:05.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:05.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:36:05.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:36:05.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 : miss 2023-12-03T11:36:05.866 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:05.867 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.864+0000 7f5b29b27800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 2023-12-03T11:36:05.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.867+0000 7f5b29b27800 -1 Injecting error=-5 at location=set_target_layout 2023-12-03T11:36:05.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.867+0000 7f5b29b27800 0 ERROR: init_target_layout failed to write target index layout to bucket info: (5) Input/output error 2023-12-03T11:36:05.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.870+0000 7f5aeb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.872+0000 7f5b29b27800 20 remove_watcher() i=0 2023-12-03T11:36:05.873 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.872+0000 7f5b29b27800 2 removed watcher, disabling cache 2023-12-03T11:36:05.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.873+0000 7f5b29b27800 20 remove_watcher() i=1 2023-12-03T11:36:05.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.874+0000 7f5b29b27800 20 remove_watcher() i=2 2023-12-03T11:36:05.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.875+0000 7f5b29b27800 20 remove_watcher() i=3 2023-12-03T11:36:05.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.876+0000 7f5b29b27800 20 remove_watcher() i=4 2023-12-03T11:36:05.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.877+0000 7f5b29b27800 20 remove_watcher() i=5 2023-12-03T11:36:05.879 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.878+0000 7f5b29b27800 20 remove_watcher() i=6 2023-12-03T11:36:05.880 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.879+0000 7f5b29b27800 20 remove_watcher() i=7 2023-12-03T11:36:05.886 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:36:05.914 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:05.914 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:05.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.927+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 realm 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.929+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.930+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.930+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.931+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:05.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.931+0000 7fd22f53e800 20 searching for the correct realm 2023-12-03T11:36:05.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.941+0000 7fd22bb04640 0 -- 172.21.15.12:0/1942155568 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x560ebff0e990 msgr2=0x560ebff10e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:05.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:05.943 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:05.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:05.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.944 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.942+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 zone default found 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 4 Realm: () 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 zonegroup default 2023-12-03T11:36:05.945 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.943+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.944+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.944+0000 7fd22f53e800 20 started sync module instance, tier type = 2023-12-03T11:36:05.946 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.944+0000 7fd22f53e800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:05.952 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.951+0000 7fd22bb04640 0 -- 172.21.15.12:0/1942155568 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7fd204079db0 msgr2=0x7fd20407c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=0 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=1 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=2 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=3 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=4 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=5 2023-12-03T11:36:05.954 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.953+0000 7fd22f53e800 20 add_watcher() i=6 2023-12-03T11:36:05.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.954+0000 7fd22f53e800 20 add_watcher() i=7 2023-12-03T11:36:05.955 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.954+0000 7fd22f53e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:05.956 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.955+0000 7fd1f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.956+0000 7fd22f53e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:05.957 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.956+0000 7fd22f53e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd22f53e800 5 note: GC not initialized 2023-12-03T11:36:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd15ffff640 20 reqs_thread_entry: start 2023-12-03T11:36:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd22f53e800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:05.963 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd22f53e800 20 Filter name: none 2023-12-03T11:36:05.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd15dffb640 20 reqs_thread_entry: start 2023-12-03T11:36:05.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.962+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.963+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:05.964 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.963+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.963+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:05.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.963+0000 7fd22f53e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.965 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.963+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.964+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.964+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.964+0000 7fd22f53e800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.964+0000 7fd22f53e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.965+0000 7fd22f53e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:05.966 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.965+0000 7fd22f53e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:05.967 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.966+0000 7fd1f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:05.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.968+0000 7fd22f53e800 20 remove_watcher() i=0 2023-12-03T11:36:05.969 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.968+0000 7fd22f53e800 2 removed watcher, disabling cache 2023-12-03T11:36:05.970 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.969+0000 7fd22f53e800 20 remove_watcher() i=1 2023-12-03T11:36:05.971 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.970+0000 7fd22f53e800 20 remove_watcher() i=2 2023-12-03T11:36:05.972 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.971+0000 7fd22f53e800 20 remove_watcher() i=3 2023-12-03T11:36:05.973 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.972+0000 7fd22f53e800 20 remove_watcher() i=4 2023-12-03T11:36:05.974 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.973+0000 7fd22f53e800 20 remove_watcher() i=5 2023-12-03T11:36:05.975 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.974+0000 7fd22f53e800 20 remove_watcher() i=6 2023-12-03T11:36:05.976 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:05.975+0000 7fd22f53e800 20 remove_watcher() i=7 2023-12-03T11:36:05.981 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:05.981 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:05.982 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:05.982 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.035 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:36:06.067 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:06.068 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:06.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.081+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 realm 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:06.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.083+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.084+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.084+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.085+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:06.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.085+0000 7fbef7233800 20 searching for the correct realm 2023-12-03T11:36:06.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.095+0000 7fbef4d33640 0 -- 172.21.15.12:0/2803493192 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x56346cfca990 msgr2=0x56346cfccd80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.096+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 zone default found 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 4 Realm: () 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:06.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:06.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 zonegroup default 2023-12-03T11:36:06.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.097+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.098+0000 7fbef7233800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.098+0000 7fbef7233800 20 started sync module instance, tier type = 2023-12-03T11:36:06.103 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.098+0000 7fbef7233800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:06.105 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.104+0000 7fbef5534640 0 -- 172.21.15.12:0/2803493192 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fbee805bc80 msgr2=0x7fbee805adf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:06.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=0 2023-12-03T11:36:06.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=1 2023-12-03T11:36:06.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=2 2023-12-03T11:36:06.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=3 2023-12-03T11:36:06.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=4 2023-12-03T11:36:06.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=5 2023-12-03T11:36:06.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.107+0000 7fbef7233800 20 add_watcher() i=6 2023-12-03T11:36:06.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.108+0000 7fbef7233800 20 add_watcher() i=7 2023-12-03T11:36:06.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.108+0000 7fbef7233800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:06.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.109+0000 7fbebafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.110+0000 7fbef7233800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:06.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.110+0000 7fbef7233800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 5 note: GC not initialized 2023-12-03T11:36:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbe29ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 20 Filter name: none 2023-12-03T11:36:06.117 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbe1b7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:06.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:36:06.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.116+0000 7fbef7233800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 : miss 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.117+0000 7fbef7233800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 2023-12-03T11:36:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18, ofs0, ns] 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbedaffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725920 cookie 94783166846512 notifier 6077 bl.length()=887 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbedaffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbedaffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbedaffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): orig_sources= new_sources= 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): potential sources added=[] removed=[] 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): orig_dests= new_dests= 2023-12-03T11:36:06.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.122+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): potential dests added=[] removed=[] 2023-12-03T11:36:06.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.124+0000 7fbef7233800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18.1.0 2023-12-03T11:36:06.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.124+0000 7fbef7233800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18.1.0 2023-12-03T11:36:06.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.124+0000 7fbef7233800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18.1.1 2023-12-03T11:36:06.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.124+0000 7fbef7233800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18.1.1 2023-12-03T11:36:06.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.124+0000 7fbef7233800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18.0 2023-12-03T11:36:06.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.126+0000 7fbef7233800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:06.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.126+0000 7fbef7233800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbef7233800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbef7233800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbef7233800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbef7233800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18, ofs0, ns] 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbedb7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725921 cookie 94783166846512 notifier 6077 bl.length()=855 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbedb7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 info.flags=0x17 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbedb7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 to cache LRU end 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.127+0000 7fbedb7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.128+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): orig_sources= new_sources= 2023-12-03T11:36:06.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.128+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): potential sources added=[] removed=[] 2023-12-03T11:36:06.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.128+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): orig_dests= new_dests= 2023-12-03T11:36:06.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.128+0000 7fbef7233800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18]): potential dests added=[] removed=[] 2023-12-03T11:36:06.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.129+0000 7fbef7233800 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2023-12-03T11:36:06.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.131+0000 7fbebafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.133+0000 7fbef7233800 20 remove_watcher() i=0 2023-12-03T11:36:06.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.133+0000 7fbef7233800 2 removed watcher, disabling cache 2023-12-03T11:36:06.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.135+0000 7fbef7233800 20 remove_watcher() i=1 2023-12-03T11:36:06.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.136+0000 7fbef7233800 20 remove_watcher() i=2 2023-12-03T11:36:06.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.137+0000 7fbef7233800 20 remove_watcher() i=3 2023-12-03T11:36:06.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.138+0000 7fbef7233800 20 remove_watcher() i=4 2023-12-03T11:36:06.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.139+0000 7fbef7233800 20 remove_watcher() i=5 2023-12-03T11:36:06.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.140+0000 7fbef7233800 20 remove_watcher() i=6 2023-12-03T11:36:06.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.141+0000 7fbef7233800 20 remove_watcher() i=7 2023-12-03T11:36:06.148 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:36:06.176 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:06.176 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:06.190 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.189+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.190+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.190+0000 7f6f7c51f800 20 realm 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.190+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.191+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.191+0000 7f6f7c51f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.191+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.191+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.192 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.191+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.192+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.193 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.192+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.193+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:06.194 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.193+0000 7f6f7c51f800 20 searching for the correct realm 2023-12-03T11:36:06.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.202+0000 7f6f78195640 0 -- 172.21.15.12:0/2240014056 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55b5e59bcb20 msgr2=0x55b5e59bef30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 20 zone default found 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 4 Realm: () 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.204+0000 7f6f7c51f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 zonegroup default 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.207 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.205+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.206+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.206+0000 7f6f7c51f800 20 started sync module instance, tier type = 2023-12-03T11:36:06.208 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.206+0000 7f6f7c51f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:06.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.211+0000 7f6f78996640 0 -- 172.21.15.12:0/2240014056 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f6f6c019b00 msgr2=0x7f6f6c045950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=0 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=1 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=2 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=3 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=4 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=5 2023-12-03T11:36:06.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.215+0000 7f6f7c51f800 20 add_watcher() i=6 2023-12-03T11:36:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.216+0000 7f6f7c51f800 20 add_watcher() i=7 2023-12-03T11:36:06.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.216+0000 7f6f7c51f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:06.218 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.217+0000 7f6f3e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.218+0000 7f6f7c51f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:06.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.218+0000 7f6f7c51f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:06.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 5 note: GC not initialized 2023-12-03T11:36:06.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6ead7fa640 20 reqs_thread_entry: start 2023-12-03T11:36:06.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:06.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 20 Filter name: none 2023-12-03T11:36:06.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6ea6ffd640 20 reqs_thread_entry: start 2023-12-03T11:36:06.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.224+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.225+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.225+0000 7f6f7c51f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.225+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.225+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.225+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.226+0000 7f6f7c51f800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.226+0000 7f6f7c51f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.226+0000 7f6f7c51f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.227+0000 7f6f7c51f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.227+0000 7f6f3e7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.229+0000 7f6f7c51f800 20 remove_watcher() i=0 2023-12-03T11:36:06.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.229+0000 7f6f7c51f800 2 removed watcher, disabling cache 2023-12-03T11:36:06.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.230+0000 7f6f7c51f800 20 remove_watcher() i=1 2023-12-03T11:36:06.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.231+0000 7f6f7c51f800 20 remove_watcher() i=2 2023-12-03T11:36:06.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.232+0000 7f6f7c51f800 20 remove_watcher() i=3 2023-12-03T11:36:06.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.233+0000 7f6f7c51f800 20 remove_watcher() i=4 2023-12-03T11:36:06.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.234+0000 7f6f7c51f800 20 remove_watcher() i=5 2023-12-03T11:36:06.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.235+0000 7f6f7c51f800 20 remove_watcher() i=6 2023-12-03T11:36:06.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.236+0000 7f6f7c51f800 20 remove_watcher() i=7 2023-12-03T11:36:06.242 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.242 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:06.242 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:36:06.242 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.327 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with ECANCELED injected at set_target_layout 2023-12-03T11:36:06.327 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.532 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:36:06.561 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:06.561 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:06.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.573+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 realm 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.575+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.576+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.576+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.577+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:06.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.577+0000 7f31c79af800 20 searching for the correct realm 2023-12-03T11:36:06.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.586+0000 7f31c54b6640 0 -- 172.21.15.12:0/2489496845 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f31bc02b040 msgr2=0x7f31bc010950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 zone default found 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 4 Realm: () 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 zonegroup default 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.588+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 started sync module instance, tier type = 2023-12-03T11:36:06.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.589+0000 7f31c79af800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:06.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.595+0000 7f31c54b6640 0 -- 172.21.15.12:0/2489496845 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f31a0079db0 msgr2=0x7f31a007c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=0 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=1 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=2 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=3 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=4 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=5 2023-12-03T11:36:06.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.598+0000 7f31c79af800 20 add_watcher() i=6 2023-12-03T11:36:06.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.599+0000 7f31c79af800 20 add_watcher() i=7 2023-12-03T11:36:06.601 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.599+0000 7f31c79af800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:06.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.601+0000 7f318afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.601+0000 7f31c79af800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:06.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.601+0000 7f31c79af800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.607+0000 7f31c79af800 5 note: GC not initialized 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.608+0000 7f30f9ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.608+0000 7f31c79af800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.608+0000 7f31c79af800 20 Filter name: none 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.608+0000 7f30eb7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:06.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.608+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.609+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.609+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.609+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.609+0000 7f31c79af800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.609+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.610+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.610+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.610+0000 7f31c79af800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.610+0000 7f31c79af800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.611+0000 7f31c79af800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.611+0000 7f31c79af800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.613 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.612+0000 7f318afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.614+0000 7f31c79af800 20 remove_watcher() i=0 2023-12-03T11:36:06.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.614+0000 7f31c79af800 2 removed watcher, disabling cache 2023-12-03T11:36:06.616 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.615+0000 7f31c79af800 20 remove_watcher() i=1 2023-12-03T11:36:06.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.616+0000 7f31c79af800 20 remove_watcher() i=2 2023-12-03T11:36:06.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.617+0000 7f31c79af800 20 remove_watcher() i=3 2023-12-03T11:36:06.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.618+0000 7f31c79af800 20 remove_watcher() i=4 2023-12-03T11:36:06.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.619+0000 7f31c79af800 20 remove_watcher() i=5 2023-12-03T11:36:06.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.620+0000 7f31c79af800 20 remove_watcher() i=6 2023-12-03T11:36:06.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.621+0000 7f31c79af800 20 remove_watcher() i=7 2023-12-03T11:36:06.627 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.627 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:06.627 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:06.627 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.629 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at set_target_layout --inject-error-code 125 2023-12-03T11:36:06.664 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:06.664 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:06.679 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.678+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 realm 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.679+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.680+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.680+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.680+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.681+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:06.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.681+0000 7f491adaf800 20 searching for the correct realm 2023-12-03T11:36:06.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.688+0000 7f49190b0640 0 -- 172.21.15.12:0/867710450 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x556bc9fa09a0 msgr2=0x556bc9fa2d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.691+0000 7f491adaf800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.691+0000 7f491adaf800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.691+0000 7f491adaf800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.691+0000 7f491adaf800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.691+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.693 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:06.694 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 zone default found 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 4 Realm: () 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 zonegroup default 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.692+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 started sync module instance, tier type = 2023-12-03T11:36:06.695 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.693+0000 7f491adaf800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:06.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.694+0000 7f49190b0640 0 -- 172.21.15.12:0/867710450 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f490c081000 msgr2=0x7f490c056650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.704 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=0 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=1 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=2 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=3 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=4 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=5 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.703+0000 7f491adaf800 20 add_watcher() i=6 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.704+0000 7f491adaf800 20 add_watcher() i=7 2023-12-03T11:36:06.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.704+0000 7f491adaf800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:06.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.706+0000 7f48defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.706+0000 7f491adaf800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:06.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.706+0000 7f491adaf800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.712+0000 7f491adaf800 5 note: GC not initialized 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.712+0000 7f484dffb640 20 reqs_thread_entry: start 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 20 Filter name: none 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f483f7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 10 cache get: name=default.rgw.meta+root+error-at-set-target-layout : miss 2023-12-03T11:36:06.714 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 10 cache put: name=default.rgw.meta+root+error-at-set-target-layout info.flags=0x11 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 10 adding default.rgw.meta+root+error-at-set-target-layout to cache LRU end 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 : miss 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.713+0000 7f491adaf800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 info.flags=0x17 2023-12-03T11:36:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 10 adding default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 to cache LRU end 2023-12-03T11:36:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.714+0000 7f491adaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 2023-12-03T11:36:06.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.717+0000 7f491adaf800 -1 Injecting error=-125 at location=set_target_layout 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f491adaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 info.flags=0x17 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f491adaf800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 to cache LRU end 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f491adaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f491adaf800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19, ofs0, ns] 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f48ff7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725924 cookie 93921438544768 notifier 6438 bl.length()=887 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f48ff7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 info.flags=0x17 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f48ff7fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 to cache LRU end 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.718+0000 7f48ff7fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.719+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): orig_sources= new_sources= 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.719+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): potential sources added=[] removed=[] 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.719+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): orig_dests= new_dests= 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.719+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): potential dests added=[] removed=[] 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.720+0000 7f491adaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19.1.0 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.720+0000 7f491adaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19.1.0 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.720+0000 7f491adaf800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19.1.1 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.720+0000 7f491adaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19.1.1 2023-12-03T11:36:06.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.720+0000 7f491adaf800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19.0 2023-12-03T11:36:06.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.722+0000 7f491adaf800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:06.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.722+0000 7f491adaf800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 2023-12-03T11:36:06.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f491adaf800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 info.flags=0x17 2023-12-03T11:36:06.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f491adaf800 10 moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 to cache LRU end 2023-12-03T11:36:06.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f491adaf800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f491adaf800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19, ofs0, ns] 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f48feffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725925 cookie 93921438544768 notifier 6438 bl.length()=855 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f48feffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 info.flags=0x17 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f48feffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 to cache LRU end 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.723+0000 7f48feffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.724+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): orig_sources= new_sources= 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.724+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): potential sources added=[] removed=[] 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.724+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): orig_dests= new_dests= 2023-12-03T11:36:06.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.724+0000 7f491adaf800 20 handle_bi_update(): bucket=:error-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19]): potential dests added=[] removed=[] 2023-12-03T11:36:06.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.725+0000 7f491adaf800 1 execute INFO: reshard of bucket "error-at-set-target-layout" completed successfully 2023-12-03T11:36:06.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.727+0000 7f48defe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.729+0000 7f491adaf800 20 remove_watcher() i=0 2023-12-03T11:36:06.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.729+0000 7f491adaf800 2 removed watcher, disabling cache 2023-12-03T11:36:06.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.730+0000 7f491adaf800 20 remove_watcher() i=1 2023-12-03T11:36:06.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.731+0000 7f491adaf800 20 remove_watcher() i=2 2023-12-03T11:36:06.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.732+0000 7f491adaf800 20 remove_watcher() i=3 2023-12-03T11:36:06.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.733+0000 7f491adaf800 20 remove_watcher() i=4 2023-12-03T11:36:06.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.734+0000 7f491adaf800 20 remove_watcher() i=5 2023-12-03T11:36:06.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.735+0000 7f491adaf800 20 remove_watcher() i=6 2023-12-03T11:36:06.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.736+0000 7f491adaf800 20 remove_watcher() i=7 2023-12-03T11:36:06.743 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-set-target-layout 2023-12-03T11:36:06.771 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:06.771 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:06.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.784+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 realm 2023-12-03T11:36:06.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:06.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.786+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.787+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.787+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.788+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:06.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.788+0000 7f026f12b800 20 searching for the correct realm 2023-12-03T11:36:06.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.797+0000 7f026ae32640 0 -- 172.21.15.12:0/2438530409 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x562c2687e270 msgr2=0x562c26880680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.798+0000 7f026f12b800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.798+0000 7f026f12b800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.798+0000 7f026f12b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.798+0000 7f026f12b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.798+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 zone default found 2023-12-03T11:36:06.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 4 Realm: () 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 zonegroup default 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.799+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.800+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.800+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.800+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.800+0000 7f026f12b800 20 started sync module instance, tier type = 2023-12-03T11:36:06.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.800+0000 7f026f12b800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:06.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.807+0000 7f026b633640 0 -- 172.21.15.12:0/2438530409 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f025401cc00 msgr2=0x7f025401a3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:06.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=0 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=1 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=2 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=3 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=4 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.809+0000 7f026f12b800 20 add_watcher() i=5 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.810+0000 7f026f12b800 20 add_watcher() i=6 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.810+0000 7f026f12b800 20 add_watcher() i=7 2023-12-03T11:36:06.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.810+0000 7f026f12b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:06.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.812+0000 7f01abfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.812+0000 7f026f12b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:06.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.812+0000 7f026f12b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.818+0000 7f026f12b800 5 note: GC not initialized 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f019affd640 20 reqs_thread_entry: start 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f026f12b800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f026f12b800 20 Filter name: none 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f0198ff9640 20 reqs_thread_entry: start 2023-12-03T11:36:06.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.819+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.820+0000 7f026f12b800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.821+0000 7f026f12b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:06.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.821+0000 7f026f12b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:06.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.822+0000 7f01abfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:06.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.824+0000 7f026f12b800 20 remove_watcher() i=0 2023-12-03T11:36:06.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.824+0000 7f026f12b800 2 removed watcher, disabling cache 2023-12-03T11:36:06.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.825+0000 7f026f12b800 20 remove_watcher() i=1 2023-12-03T11:36:06.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.826+0000 7f026f12b800 20 remove_watcher() i=2 2023-12-03T11:36:06.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.828+0000 7f026f12b800 20 remove_watcher() i=3 2023-12-03T11:36:06.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.829+0000 7f026f12b800 20 remove_watcher() i=4 2023-12-03T11:36:06.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.830+0000 7f026f12b800 20 remove_watcher() i=5 2023-12-03T11:36:06.832 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.831+0000 7f026f12b800 20 remove_watcher() i=6 2023-12-03T11:36:06.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:06.832+0000 7f026f12b800 20 remove_watcher() i=7 2023-12-03T11:36:06.837 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.837 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:06.838 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19 num_objects: 20 size_kb: 1 num_shards: 2 2023-12-03T11:36:06.838 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:06.953 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with abort at set_target_layout 2023-12-03T11:36:06.953 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.169 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:36:07.196 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:07.196 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:07.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.209+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 realm 2023-12-03T11:36:07.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.211+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.212+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.212+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.213+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.213+0000 7f86d3f24800 20 searching for the correct realm 2023-12-03T11:36:07.223 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.222+0000 7f86cfbc9640 0 -- 172.21.15.12:0/3643807064 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55574e658470 msgr2=0x55574e65a860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.223+0000 7f86d3f24800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.223+0000 7f86d3f24800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.223+0000 7f86d3f24800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.223+0000 7f86d3f24800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.223+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 zone default found 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 4 Realm: () 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 zonegroup default 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.224+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 started sync module instance, tier type = 2023-12-03T11:36:07.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.225+0000 7f86d3f24800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.232+0000 7f86d03ca640 0 -- 172.21.15.12:0/3643807064 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f86c4058680 msgr2=0x7f86c4087210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=0 2023-12-03T11:36:07.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=1 2023-12-03T11:36:07.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=2 2023-12-03T11:36:07.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=3 2023-12-03T11:36:07.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=4 2023-12-03T11:36:07.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=5 2023-12-03T11:36:07.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.234+0000 7f86d3f24800 20 add_watcher() i=6 2023-12-03T11:36:07.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.236+0000 7f86d3f24800 20 add_watcher() i=7 2023-12-03T11:36:07.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.236+0000 7f86d3f24800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.237+0000 7f8695fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.238+0000 7f86d3f24800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.239 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.238+0000 7f86d3f24800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f86d3f24800 5 note: GC not initialized 2023-12-03T11:36:07.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f860cff9640 20 reqs_thread_entry: start 2023-12-03T11:36:07.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f86d3f24800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f86d3f24800 20 Filter name: none 2023-12-03T11:36:07.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f86027fc640 20 reqs_thread_entry: start 2023-12-03T11:36:07.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.244+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.245+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.245+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.245+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.245+0000 7f86d3f24800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.245+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.246+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.246+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.246+0000 7f86d3f24800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.246+0000 7f86d3f24800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:07.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.247+0000 7f86d3f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.247+0000 7f86d3f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.249 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.248+0000 7f8695fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.250+0000 7f86d3f24800 20 remove_watcher() i=0 2023-12-03T11:36:07.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.250+0000 7f86d3f24800 2 removed watcher, disabling cache 2023-12-03T11:36:07.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.252+0000 7f86d3f24800 20 remove_watcher() i=1 2023-12-03T11:36:07.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.254+0000 7f86d3f24800 20 remove_watcher() i=2 2023-12-03T11:36:07.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.255+0000 7f86d3f24800 20 remove_watcher() i=3 2023-12-03T11:36:07.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.256+0000 7f86d3f24800 20 remove_watcher() i=4 2023-12-03T11:36:07.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.257+0000 7f86d3f24800 20 remove_watcher() i=5 2023-12-03T11:36:07.259 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.258+0000 7f86d3f24800 20 remove_watcher() i=6 2023-12-03T11:36:07.261 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.260+0000 7f86d3f24800 20 remove_watcher() i=7 2023-12-03T11:36:07.265 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.265 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:07.266 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:07.266 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.267 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at set_target_layout 2023-12-03T11:36:07.300 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:07.300 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:07.314 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.313+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 realm 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.317+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.317+0000 7f1a9a137800 20 searching for the correct realm 2023-12-03T11:36:07.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.326+0000 7f1a95ecd640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a755906600 msgr2=0x55a755908a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zone default found 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Realm: () 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zonegroup default 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.332 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started sync module instance, tier type = 2023-12-03T11:36:07.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.335+0000 7f1a966ce640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55a75590e750 msgr2=0x55a755910c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=0 2023-12-03T11:36:07.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=1 2023-12-03T11:36:07.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=2 2023-12-03T11:36:07.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=3 2023-12-03T11:36:07.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=4 2023-12-03T11:36:07.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=5 2023-12-03T11:36:07.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=6 2023-12-03T11:36:07.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.339+0000 7f1a9a137800 20 add_watcher() i=7 2023-12-03T11:36:07.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.339+0000 7f1a9a137800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.340+0000 7f1a5b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.347+0000 7f1a9a137800 5 note: GC not initialized 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.347+0000 7f19ca7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.347+0000 7f1a9a137800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.347+0000 7f1a9a137800 20 Filter name: none 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f19bbfff640 20 reqs_thread_entry: start 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:36:07.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : miss 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:07.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.349+0000 7f1a9a137800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:07.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f1a9a137800 time 2023-12-03T11:36:07.353597+0000 2023-12-03T11:36:07.353 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:07.354 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.354 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f1a9cd6eef8] 2023-12-03T11:36:07.354 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.354 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.354 INFO:tasks.workunit.client.0.smithi012.stderr: 4: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 6: main() 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 8: __libc_start_main() 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 9: _start() 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.353+0000 7f1a9a137800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f1a9a137800 time 2023-12-03T11:36:07.353597+0000 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f1a9cd6eef8] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 4: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.355 INFO:tasks.workunit.client.0.smithi012.stderr: 6: main() 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr: 8: __libc_start_main() 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr: 9: _start() 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:36:07.356 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f1a9a137800 thread_name:radosgw-admin 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1a9a654db0] 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f1a9a6a154c] 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1a9cd6ef52] 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.372 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 8: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 10: main() 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 12: __libc_start_main() 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 13: _start() 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.354+0000 7f1a9a137800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f1a9a137800 thread_name:radosgw-admin 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1a9a654db0] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f1a9a6a154c] 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:07.373 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1a9cd6ef52] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 8: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 10: main() 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 12: __libc_start_main() 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 13: _start() 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:36:07.374 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:36:07.307+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:36:07.307+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: dump: 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.377 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:36:07.307+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6501 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:36:07.307+0000 7f1a87fff640 4 set_mon_vals no callback set 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: dump: 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.378 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:36:07.308+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: get_monmap_and_config 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:36:07.308+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.379 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.380 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:36:07.308+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.381 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.382 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:36:07.309+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:36:07.309+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x7fff2af1b488 EEXIST 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a7558e4f10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a7558f0430 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a7558e7950 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.492149s 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a7558f0430 auth_method 0 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.383 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a7558e4f10 auth_method 0 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a7558e7950 auth_method 0 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.384 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_done global_id 6504 payload 274 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:36:07.310+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f1a78069820 auth_method 0 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: dump: 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.385 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: dump: 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.386 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6504 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:36:07.310+0000 7f1a9a137800 4 set_mon_vals no callback set 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: shutdown 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:36:07.310+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.387 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.388 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: starting msgr at 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: starting objecter 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command objecter_requests cmddesc objecter_requests hook 0x55a7558f0850 EEXIST 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: setting wanted keys 2023-12-03T11:36:07.389 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: calling monclient init 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.390 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x55a7558b4900 EEXIST 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a7558e4f10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a7558e7950 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a7558f0430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.494482s 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a7558e4f10 auth_method 0 2023-12-03T11:36:07.391 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a7558f0430 auth_method 0 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a7558e7950 auth_method 0 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.392 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_done global_id 6507 payload 824 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: dump: 2023-12-03T11:36:07.393 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.394 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: dump: 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:36:07.312+0000 7f1a77fff640 4 set_mon_vals no callback set 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:36:07.312+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6507 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.395 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:36:07.312+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:36:07.313+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:36:07.314+0000 7f1a956cc640 10 monclient: get_auth_request con 0x7f1a70069820 auth_method 0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:36:07.314+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a755902880 auth_method 0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 realm 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:36:07.315+0000 7f1a9a137800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.396 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:36:07.315+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x55a755906600 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:36:07.316+0000 7f1a956cc640 10 monclient: get_auth_request con 0x55a755909df0 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:36:07.317+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:36:07.317+0000 7f1a9a137800 20 searching for the correct realm 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:36:07.317+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a75590e750 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:36:07.318+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a8c020070 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:36:07.319+0000 7f1a956cc640 10 monclient: get_auth_request con 0x7f1a88056a20 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:36:07.320+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f1a8c034960 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:36:07.323+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a8c015420 auth_method 0 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:36:07.326+0000 7f1a95ecd640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a755906600 msgr2=0x55a755908a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.397 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:36:07.327+0000 7f1a767fc640 1 client.6507.objecter ms_handle_reset 0x55a755906600 session 0x55a755675320 osd.5 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:36:07.327+0000 7f1a767fc640 10 monclient: _renew_subs 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:36:07.327+0000 7f1a767fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:36:07.327+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a70079d80 auth_method 0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.398 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zone default found 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Realm: () 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:36:07.328+0000 7f1a9a137800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zonegroup default 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started sync module instance, tier type = 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:36:07.335+0000 7f1a966ce640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55a75590e750 msgr2=0x55a755910c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.399 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:36:07.335+0000 7f1a767fc640 1 client.6507.objecter ms_handle_reset 0x55a75590e750 session 0x55a75590e0d0 osd.3 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:36:07.336+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a755906600 auth_method 0 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=0 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=1 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=2 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=3 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=4 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=5 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=6 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:36:07.339+0000 7f1a9a137800 20 add_watcher() i=7 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:36:07.339+0000 7f1a9a137800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.400 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:36:07.340+0000 7f1a5b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:36:07.341+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.401 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache list hook 0x55a75589c630 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache inspect hook 0x55a75589c630 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache erase hook 0x55a75589c630 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache zap hook 0x55a75589c630 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: get_monmap_and_config 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:36:07.341+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.402 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.403 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:36:07.341+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.404 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:36:07.342+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:36:07.342+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:36:07.342+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x7fff2af1ba08 EEXIST 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a75594c530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a755948820 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a75594bdb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.405 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.524955s 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a75594c530 auth_method 0 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a75594bdb0 auth_method 0 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.406 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a755948820 auth_method 0 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_done global_id 6510 payload 274 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:36:07.342+0000 7f1a966ce640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.407 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: dump: 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.408 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: dump: 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6510 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:07.409 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:36:07.343+0000 7f1a9a137800 4 set_mon_vals no callback set 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: shutdown 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:36:07.343+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.410 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.411 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: starting msgr at 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: starting objecter 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command objecter_requests cmddesc objecter_requests hook 0x55a755742740 EEXIST 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: setting wanted keys 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: calling monclient init 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.412 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x55a75590f8f0 EEXIST 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a755948820 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a75594bdb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.413 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a75594c530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.527529s 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a75594c530 auth_method 0 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a75594bdb0 auth_method 0 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a755948820 auth_method 0 2023-12-03T11:36:07.414 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_done global_id 6200 payload 824 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.415 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:36:07.345+0000 7f1a956cc640 1 monclient: found mon.noname-c 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: dump: 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.416 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: dump: 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.417 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:36:07.345+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6200 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:36:07.345+0000 7f1a5afe5640 4 set_mon_vals no callback set 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:36:07.345+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:36:07.345+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cr dump hook 0x55a75595e3d8 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:36:07.347+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f19d0069820 auth_method 0 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 note: GC not initialized 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:36:07.347+0000 7f19ca7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace show hook 0x7f1a90004720 2023-12-03T11:36:07.418 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace history hook 0x7f1a90004720 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace active hook 0x7f1a90004720 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace active_short hook 0x7f1a90004720 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:36:07.347+0000 7f1a9a137800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:36:07.347+0000 7f1a9a137800 20 Filter name: none 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:36:07.348+0000 7f19bbfff640 20 reqs_thread_entry: start 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : miss 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:36:07.349+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:07.419 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:36:07.349+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x55a755999f90 auth_method 0 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:36:07.353+0000 7f1a9a137800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f1a9a137800 time 2023-12-03T11:36:07.353597+0000 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f1a9cd6eef8] 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 4: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 6: main() 2023-12-03T11:36:07.420 INFO:tasks.workunit.client.0.smithi012.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 8: __libc_start_main() 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 9: _start() 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:36:07.354+0000 7f1a9a137800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f1a9a137800 thread_name:radosgw-admin 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1a9a654db0] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f1a9a6a154c] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1a9cd6ef52] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 8: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.421 INFO:tasks.workunit.client.0.smithi012.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 10: main() 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 12: __libc_start_main() 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 13: _start() 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:36:07.422 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:36:07.423 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:36:07.424 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:36:07.425 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:36:07.426 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:36:07.427 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f19bbfff640 / http_manager 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f19ca7fc640 / http_manager 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a597e2640 / ms_dispatch 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a5afe5640 / io_context_pool 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a5b7e6640 / rgw_dt_lg_renew 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a767fc640 / ms_dispatch 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a77fff640 / io_context_pool 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a86ffd640 / ms_dispatch 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a87fff640 / io_context_pool 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a956cc640 / msgr-worker-2 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a95ecd640 / msgr-worker-1 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a966ce640 / msgr-worker-0 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a9a137800 / radosgw-admin 2023-12-03T11:36:07.428 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/log/ceph/ceph-client.admin.101439.log 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:36:07.307+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:36:07.307+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: dump: 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.429 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:36:07.307+0000 7f1a86ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:36:07.307+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6501 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:36:07.307+0000 7f1a87fff640 4 set_mon_vals no callback set 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:36:07.308+0000 7f1a86ffd640 10 monclient: dump: 2023-12-03T11:36:07.430 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:36:07.308+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: get_monmap_and_config 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:36:07.308+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.431 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:36:07.308+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding auth protocol: cephx 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.432 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: crc 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:36:07.308+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b61b0) adding con mode: secure 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:36:07.308+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.433 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding auth protocol: cephx 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: crc 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b340) adding con mode: secure 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:36:07.309+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:36:07.309+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.434 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:36:07.309+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x7fff2af1b488 EEXIST 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a7558e4f10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a7558f0430 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a7558e7950 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:36:07.309+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.492149s 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a7558f0430 auth_method 0 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.435 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a7558e4f10 auth_method 0 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:36:07.309+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a7558e7950 auth_method 0 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.436 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:36:07.309+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_done global_id 6504 payload 274 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:36:07.309+0000 7f1a86ffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:36:07.309+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:36:07.310+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f1a78069820 auth_method 0 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: dump: 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.437 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:36:07.310+0000 7f1a77fff640 10 monclient: dump: 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.438 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6504 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:36:07.310+0000 7f1a9a137800 4 set_mon_vals no callback set 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: shutdown 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:36:07.310+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.439 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:36:07.310+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding auth protocol: cephx 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.440 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: crc 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:36:07.310+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b6080) adding con mode: secure 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: starting msgr at 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: starting objecter 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command objecter_requests cmddesc objecter_requests hook 0x55a7558f0850 EEXIST 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: setting wanted keys 2023-12-03T11:36:07.441 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:36:07.311+0000 7f1a9a137800 1 librados: calling monclient init 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding auth protocol: cephx 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.442 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: crc 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 AuthRegistry(0x55a7558b47b8) adding con mode: secure 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:36:07.311+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:36:07.311+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x55a7558b4900 EEXIST 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a7558e4f10 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a7558e7950 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a7558f0430 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:36:07.311+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.494482s 2023-12-03T11:36:07.443 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a7558e4f10 auth_method 0 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:36:07.311+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a7558f0430 auth_method 0 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a7558e7950 auth_method 0 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.444 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:36:07.312+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:36:07.312+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_done global_id 6507 payload 824 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:36:07.312+0000 7f1a95ecd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: dump: 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.445 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:36:07.312+0000 7f1a767fc640 10 monclient: dump: 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.446 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:36:07.312+0000 7f1a77fff640 4 set_mon_vals no callback set 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:36:07.312+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6507 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:36:07.312+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.447 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:36:07.312+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:36:07.313+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:36:07.314+0000 7f1a767fc640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:36:07.314+0000 7f1a956cc640 10 monclient: get_auth_request con 0x7f1a70069820 auth_method 0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:36:07.314+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a755902880 auth_method 0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 realm 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:36:07.315+0000 7f1a9a137800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:36:07.315+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:36:07.315+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x55a755906600 auth_method 0 2023-12-03T11:36:07.448 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:36:07.316+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:36:07.316+0000 7f1a956cc640 10 monclient: get_auth_request con 0x55a755909df0 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:36:07.317+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:36:07.317+0000 7f1a9a137800 20 searching for the correct realm 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:36:07.317+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a75590e750 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:36:07.318+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a8c020070 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:36:07.319+0000 7f1a956cc640 10 monclient: get_auth_request con 0x7f1a88056a20 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:36:07.320+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f1a8c034960 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:36:07.323+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a8c015420 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:36:07.326+0000 7f1a95ecd640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55a755906600 msgr2=0x55a755908a10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:36:07.327+0000 7f1a767fc640 1 client.6507.objecter ms_handle_reset 0x55a755906600 session 0x55a755675320 osd.5 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:36:07.327+0000 7f1a767fc640 10 monclient: _renew_subs 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:36:07.327+0000 7f1a767fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:36:07.327+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x7f1a70079d80 auth_method 0 2023-12-03T11:36:07.449 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zone default found 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Realm: () 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:36:07.328+0000 7f1a9a137800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:36:07.328+0000 7f1a9a137800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.450 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 zonegroup default 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:36:07.328+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started sync module instance, tier type = 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:36:07.329+0000 7f1a9a137800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:36:07.335+0000 7f1a966ce640 0 -- 172.21.15.12:0/3965050716 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55a75590e750 msgr2=0x55a755910c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:36:07.335+0000 7f1a767fc640 1 client.6507.objecter ms_handle_reset 0x55a75590e750 session 0x55a75590e0d0 osd.3 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:36:07.336+0000 7f1a966ce640 10 monclient: get_auth_request con 0x55a755906600 auth_method 0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=0 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=1 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=2 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=3 2023-12-03T11:36:07.451 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=4 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=5 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:36:07.338+0000 7f1a9a137800 20 add_watcher() i=6 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:36:07.339+0000 7f1a9a137800 20 add_watcher() i=7 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:36:07.339+0000 7f1a9a137800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding auth protocol: cephx 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:36:07.340+0000 7f1a5b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.452 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: crc 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:36:07.340+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1bd70) adding con mode: secure 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:36:07.341+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:36:07.341+0000 7f1a9a137800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache list hook 0x55a75589c630 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache inspect hook 0x55a75589c630 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache erase hook 0x55a75589c630 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cache zap hook 0x55a75589c630 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: get_monmap_and_config 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.453 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:36:07.341+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding auth protocol: cephx 2023-12-03T11:36:07.454 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: crc 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x55a75591c740) adding con mode: secure 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:36:07.341+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:36:07.341+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.455 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding auth protocol: cephx 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: crc 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:36:07.341+0000 7f1a9a137800 5 AuthRegistry(0x7fff2af1b8c0) adding con mode: secure 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:36:07.342+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:36:07.342+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:36:07.342+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x7fff2af1ba08 EEXIST 2023-12-03T11:36:07.456 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient: _add_conns ranks=[0,1,2] 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a75594c530 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a755948820 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a75594bdb0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:36:07.342+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.524955s 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a75594c530 auth_method 0 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a75594bdb0 auth_method 0 2023-12-03T11:36:07.457 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:36:07.342+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a755948820 auth_method 0 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.458 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:36:07.342+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient(hunting): handle_auth_done global_id 6510 payload 274 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:36:07.342+0000 7f1a966ce640 1 monclient: found mon.noname-a 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:36:07.342+0000 7f1a966ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: dump: 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.459 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:36:07.343+0000 7f1a5afe5640 10 monclient: dump: 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.460 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6510 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:36:07.343+0000 7f1a9a137800 4 set_mon_vals no callback set 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: shutdown 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: build_initial_monmap 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:36:07.343+0000 7f1a9a137800 1 build_initial for_mkfs: 0 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:36:07.343+0000 7f1a9a137800 10 monclient: monmap: 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.461 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding auth protocol: cephx 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.462 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: crc 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:36:07.343+0000 7f1a9a137800 5 AuthRegistry(0x55a7559428e0) adding con mode: secure 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: starting msgr at 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: starting objecter 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command objecter_requests cmddesc objecter_requests hook 0x55a755742740 EEXIST 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: setting wanted keys 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:36:07.344+0000 7f1a9a137800 1 librados: calling monclient init 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: init 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding auth protocol: cephx 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.463 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: crc 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 AuthRegistry(0x55a75590f7a8) adding con mode: secure 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:36:07.344+0000 7f1a9a137800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:36:07.344+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command rotate-key cmddesc rotate-key hook 0x55a75590f8f0 EEXIST 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:36:07.464 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-b con 0x55a755948820 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-a con 0x55a75594bdb0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): picked mon.noname-c con 0x55a75594c530 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:36:07.344+0000 7f1a9a137800 10 monclient(hunting): authenticate will time out at 1577.527529s 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): get_auth_request con 0x55a75594c530 auth_method 0 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request con 0x55a75594bdb0 auth_method 0 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.465 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): get_auth_request con 0x55a755948820 auth_method 0 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:36:07.345+0000 7f1a95ecd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:36:07.345+0000 7f1a966ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:07.466 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient(hunting): handle_auth_done global_id 6200 payload 824 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient: _finish_hunting 0 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:36:07.345+0000 7f1a956cc640 1 monclient: found mon.noname-c 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:36:07.345+0000 7f1a956cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: dump: 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.467 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: _finish_auth 0 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: _check_auth_tickets 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:36:07.345+0000 7f1a597e2640 10 monclient: dump: 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:07.468 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:36:07.345+0000 7f1a9a137800 5 monclient: authenticate success, global_id 6200 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:36:07.345+0000 7f1a5afe5640 4 set_mon_vals no callback set 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _renew_subs 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:36:07.345+0000 7f1a9a137800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:36:07.345+0000 7f1a9a137800 1 librados: init done 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:36:07.345+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command cr dump hook 0x55a75595e3d8 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:36:07.347+0000 7f1a597e2640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:07.469 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:36:07.347+0000 7f1a966ce640 10 monclient: get_auth_request con 0x7f19d0069820 auth_method 0 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 note: GC not initialized 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:36:07.347+0000 7f19ca7fc640 20 reqs_thread_entry: start 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace show hook 0x7f1a90004720 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace history hook 0x7f1a90004720 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace active hook 0x7f1a90004720 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:36:07.347+0000 7f1a9a137800 5 asok(0x55a7556aca10) register_command sync trace active_short hook 0x7f1a90004720 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:36:07.347+0000 7f1a9a137800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:36:07.347+0000 7f1a9a137800 20 Filter name: none 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:36:07.348+0000 7f19bbfff640 20 reqs_thread_entry: start 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:36:07.470 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:36:07.348+0000 7f1a9a137800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : miss 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:36:07.348+0000 7f1a9a137800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:36:07.349+0000 7f1a9a137800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:36:07.349+0000 7f1a9a137800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:36:07.349+0000 7f1a95ecd640 10 monclient: get_auth_request con 0x55a755999f90 auth_method 0 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:36:07.353+0000 7f1a9a137800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f1a9a137800 time 2023-12-03T11:36:07.353597+0000 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f1a9cd6eef8] 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.471 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 4: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 6: main() 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 7: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 8: __libc_start_main() 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 9: _start() 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:36:07.354+0000 7f1a9a137800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f1a9a137800 thread_name:radosgw-admin 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f1a9a654db0] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f1a9a6a154c] 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:07.472 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f1a9cd6ef52] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f1a9cd6f0b6] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55a754be3a71] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 8: radosgw-admin(+0x79787a) [0x55a754beb87a] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x24b) [0x55a754befe6b] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 10: main() 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 11: /lib64/libc.so.6(+0x3feb0) [0x7f1a9a63feb0] 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 12: __libc_start_main() 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 13: _start() 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:36:07.473 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:36:07.474 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:36:07.475 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:36:07.476 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:36:07.477 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:36:07.478 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f19bbfff640 / http_manager 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f19ca7fc640 / http_manager 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a597e2640 / ms_dispatch 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a5afe5640 / io_context_pool 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a5b7e6640 / rgw_dt_lg_renew 2023-12-03T11:36:07.479 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a767fc640 / ms_dispatch 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a77fff640 / io_context_pool 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a86ffd640 / ms_dispatch 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a87fff640 / io_context_pool 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a956cc640 / msgr-worker-2 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a95ecd640 / msgr-worker-1 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a966ce640 / msgr-worker-0 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1a9a137800 / radosgw-admin 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:36:07.355174Z_b76127ab-5e18-483c-98aa-2d673892874d/log 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:36:07.480 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.402+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 realm 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.404+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.405+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.405+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.406+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.406+0000 7efdab8c0800 20 searching for the correct realm 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.413+0000 7efda8bbf640 0 -- 172.21.15.12:0/1361724385 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x56312d077ff0 msgr2=0x56312d07a3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.417+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 zone default found 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 4 Realm: () 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 zonegroup default 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efda93c0640 0 -- 172.21.15.12:0/1361724385 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7efda0059600 msgr2=0x7efda0057800 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.418+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.419+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.419+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.419+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.419+0000 7efdab8c0800 20 started sync module instance, tier type = 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.419+0000 7efdab8c0800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=0 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=1 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=2 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=3 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=4 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=5 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.428+0000 7efdab8c0800 20 add_watcher() i=6 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.429+0000 7efdab8c0800 20 add_watcher() i=7 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.429+0000 7efdab8c0800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.430+0000 7efd6efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.431+0000 7efdab8c0800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.431+0000 7efdab8c0800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.436+0000 7efdab8c0800 5 note: GC not initialized 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.437+0000 7efcddffb640 20 reqs_thread_entry: start 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.437+0000 7efdab8c0800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.437+0000 7efdab8c0800 20 Filter name: none 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.437+0000 7efccf7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.437+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.438+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.438+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.438+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.438+0000 7efdab8c0800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.438+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.439+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.439+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.439+0000 7efdab8c0800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.439+0000 7efdab8c0800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:07.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.439+0000 7efdab8c0800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.440+0000 7efdab8c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.440+0000 7efd6efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.443+0000 7efdab8c0800 20 remove_watcher() i=0 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.443+0000 7efdab8c0800 2 removed watcher, disabling cache 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.444+0000 7efdab8c0800 20 remove_watcher() i=1 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.445+0000 7efdab8c0800 20 remove_watcher() i=2 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.446+0000 7efdab8c0800 20 remove_watcher() i=3 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.447+0000 7efdab8c0800 20 remove_watcher() i=4 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.448+0000 7efdab8c0800 20 remove_watcher() i=5 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.449+0000 7efdab8c0800 20 remove_watcher() i=6 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.450+0000 7efdab8c0800 20 remove_watcher() i=7 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:07.490 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:07.508 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:36:07.539 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:07.539 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:07.553 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.552+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 realm 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.554+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.555+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.556 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.555+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.556+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:07.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.556+0000 7ff008d30800 20 searching for the correct realm 2023-12-03T11:36:07.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.564+0000 7ff004ac6640 0 -- 172.21.15.12:0/1549508481 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7feff8074470 msgr2=0x7feff805c120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:07.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:07.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:07.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:07.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.566+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 zone default found 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 4 Realm: () 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 zonegroup default 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.567+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.568+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.568+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.568+0000 7ff008d30800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.568+0000 7ff008d30800 20 started sync module instance, tier type = 2023-12-03T11:36:07.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.568+0000 7ff008d30800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:07.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.573+0000 7ff0042c5640 0 -- 172.21.15.12:0/1549508481 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55fe4aaa15e0 msgr2=0x55fe4aaa39d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:07.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=0 2023-12-03T11:36:07.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=1 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=2 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=3 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=4 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=5 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.577+0000 7ff008d30800 20 add_watcher() i=6 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.578+0000 7ff008d30800 20 add_watcher() i=7 2023-12-03T11:36:07.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.578+0000 7ff008d30800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:07.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.579+0000 7fefcafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.580+0000 7ff008d30800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:07.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.580+0000 7ff008d30800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:07.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7ff008d30800 5 note: GC not initialized 2023-12-03T11:36:07.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7fef39ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:07.587 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7ff008d30800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7ff008d30800 20 Filter name: none 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7fef2b7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7ff008d30800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.586+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:07.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : miss 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 20 rados->read ofs=0 len=0 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:07.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.587+0000 7ff008d30800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:07.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.589+0000 7ff008d30800 0 INFO: RGWReshardLock::lock found lock on abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to be held by another RGW process; skipping for now 2023-12-03T11:36:07.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.590+0000 7fefcafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:07.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.592+0000 7ff008d30800 20 remove_watcher() i=0 2023-12-03T11:36:07.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.592+0000 7ff008d30800 2 removed watcher, disabling cache 2023-12-03T11:36:07.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.593+0000 7ff008d30800 20 remove_watcher() i=1 2023-12-03T11:36:07.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.594+0000 7ff008d30800 20 remove_watcher() i=2 2023-12-03T11:36:07.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.595+0000 7ff008d30800 20 remove_watcher() i=3 2023-12-03T11:36:07.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.596+0000 7ff008d30800 20 remove_watcher() i=4 2023-12-03T11:36:07.598 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.597+0000 7ff008d30800 20 remove_watcher() i=5 2023-12-03T11:36:07.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.598+0000 7ff008d30800 20 remove_watcher() i=6 2023-12-03T11:36:07.600 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:07.599+0000 7ff008d30800 20 remove_watcher() i=7 2023-12-03T11:36:07.605 INFO:tasks.workunit.client.0.smithi012.stderr:waiting 30 seconds for reshard lock to expire... 2023-12-03T11:36:37.632 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-set-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:36:37.667 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:37.667 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:37.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.683+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 realm 2023-12-03T11:36:37.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.685+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.686+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:37.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.686+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.688+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:37.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.688+0000 7f17d592b800 20 searching for the correct realm 2023-12-03T11:36:37.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.699+0000 7f17d1639640 0 -- 172.21.15.12:0/2657117614 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55ffd15ed880 msgr2=0x55ffd15efc70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.703 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.702+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 zone default found 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 4 Realm: () 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 zonegroup default 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.703+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.704+0000 7f17d592b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.704+0000 7f17d592b800 20 started sync module instance, tier type = 2023-12-03T11:36:37.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.704+0000 7f17d592b800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:37.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.710+0000 7f17d1e3a640 0 -- 172.21.15.12:0/2657117614 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f17c807d040 msgr2=0x7f17c8089a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=0 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=1 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=2 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=3 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=4 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=5 2023-12-03T11:36:37.715 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.714+0000 7f17d592b800 20 add_watcher() i=6 2023-12-03T11:36:37.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.715+0000 7f17d592b800 20 add_watcher() i=7 2023-12-03T11:36:37.716 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.715+0000 7f17d592b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:37.717 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.716+0000 7f1796fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:37.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.717+0000 7f17d592b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:37.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.717+0000 7f17d592b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:37.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.723+0000 7f17d592b800 5 note: GC not initialized 2023-12-03T11:36:37.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.723+0000 7f1705ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:37.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:37.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 20 Filter name: none 2023-12-03T11:36:37.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f16f77fe640 20 reqs_thread_entry: start 2023-12-03T11:36:37.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 10 cache get: name=default.rgw.meta+root+abort-at-set-target-layout : miss 2023-12-03T11:36:37.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:37.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 10 cache put: name=default.rgw.meta+root+abort-at-set-target-layout info.flags=0x11 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 10 adding default.rgw.meta+root+abort-at-set-target-layout to cache LRU end 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : miss 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.724+0000 7f17d592b800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:37.727 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.725+0000 7f17d592b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17d592b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17d592b800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17d592b800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17d592b800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20, ofs0, ns] 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17b37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725964 cookie 94557217726992 notifier 6248 bl.length()=887 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17b37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:37.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17b37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.730+0000 7f17b37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.731+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): orig_sources= new_sources= 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.731+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): potential sources added=[] removed=[] 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.731+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): orig_dests= new_dests= 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.731+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): potential dests added=[] removed=[] 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.732+0000 7f17d592b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20.1.0 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.732+0000 7f17d592b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20.1.0 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.732+0000 7f17d592b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20.1.1 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.732+0000 7f17d592b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20.1.1 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.732+0000 7f17d592b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20.0 2023-12-03T11:36:37.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.734+0000 7f17d592b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:37.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.734+0000 7f17d592b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 2023-12-03T11:36:37.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17d592b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:37.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17d592b800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:37.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17d592b800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:37.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17d592b800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20, ofs0, ns] 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17b2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725965 cookie 94557217726992 notifier 6248 bl.length()=855 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17b2ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 info.flags=0x17 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17b2ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-set-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 to cache LRU end 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.736+0000 7f17b2ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.737+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): orig_sources= new_sources= 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.737+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): potential sources added=[] removed=[] 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.737+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): orig_dests= new_dests= 2023-12-03T11:36:37.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.737+0000 7f17d592b800 20 handle_bi_update(): bucket=:abort-at-set-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20]): potential dests added=[] removed=[] 2023-12-03T11:36:37.739 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.738+0000 7f17d592b800 1 execute INFO: reshard of bucket "abort-at-set-target-layout" completed successfully 2023-12-03T11:36:37.741 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.740+0000 7f1796fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:37.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.742+0000 7f17d592b800 20 remove_watcher() i=0 2023-12-03T11:36:37.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.742+0000 7f17d592b800 2 removed watcher, disabling cache 2023-12-03T11:36:37.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.744+0000 7f17d592b800 20 remove_watcher() i=1 2023-12-03T11:36:37.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.745+0000 7f17d592b800 20 remove_watcher() i=2 2023-12-03T11:36:37.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.746+0000 7f17d592b800 20 remove_watcher() i=3 2023-12-03T11:36:37.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.747+0000 7f17d592b800 20 remove_watcher() i=4 2023-12-03T11:36:37.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.748+0000 7f17d592b800 20 remove_watcher() i=5 2023-12-03T11:36:37.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.749+0000 7f17d592b800 20 remove_watcher() i=6 2023-12-03T11:36:37.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.750+0000 7f17d592b800 20 remove_watcher() i=7 2023-12-03T11:36:37.758 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-set-target-layout 2023-12-03T11:36:37.786 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:37.786 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:37.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.799+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 20 realm 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.801+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.802+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.802+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.803+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:37.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.803+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.804+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:37.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.804+0000 7f17c833a800 20 searching for the correct realm 2023-12-03T11:36:37.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.813+0000 7f17c400d640 0 -- 172.21.15.12:0/344127065 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f17b8084d50 msgr2=0x7f17b80847c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.815+0000 7f17c833a800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.815+0000 7f17c833a800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.815+0000 7f17c833a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.815+0000 7f17c833a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.815+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:37.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 zone default found 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 4 Realm: () 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 zonegroup default 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.816+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.819 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 started sync module instance, tier type = 2023-12-03T11:36:37.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.817+0000 7f17c833a800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:37.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.822+0000 7f17c380c640 0 -- 172.21.15.12:0/344127065 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55c82912e780 msgr2=0x55c829130b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=0 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=1 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=2 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=3 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=4 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=5 2023-12-03T11:36:37.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.827+0000 7f17c833a800 20 add_watcher() i=6 2023-12-03T11:36:37.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.828+0000 7f17c833a800 20 add_watcher() i=7 2023-12-03T11:36:37.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.828+0000 7f17c833a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:37.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.829+0000 7f178a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:37.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.830+0000 7f17c833a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:37.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.830+0000 7f17c833a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:37.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f17c833a800 5 note: GC not initialized 2023-12-03T11:36:37.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f16f97fa640 20 reqs_thread_entry: start 2023-12-03T11:36:37.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f17c833a800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:37.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f17c833a800 20 Filter name: none 2023-12-03T11:36:37.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f16f2ffd640 20 reqs_thread_entry: start 2023-12-03T11:36:37.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.836+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.837+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.837+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.837+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.837+0000 7f17c833a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.837+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.838+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=194 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.838+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.838+0000 7f17c833a800 20 rados_obj.operate() r=0 bl.length=378 2023-12-03T11:36:37.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.838+0000 7f17c833a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.838+0000 7f17c833a800 20 rados->read ofs=0 len=0 2023-12-03T11:36:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.839+0000 7f17c833a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:37.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.839+0000 7f178a7e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:37.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.842+0000 7f17c833a800 20 remove_watcher() i=0 2023-12-03T11:36:37.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.842+0000 7f17c833a800 2 removed watcher, disabling cache 2023-12-03T11:36:37.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.843+0000 7f17c833a800 20 remove_watcher() i=1 2023-12-03T11:36:37.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.844+0000 7f17c833a800 20 remove_watcher() i=2 2023-12-03T11:36:37.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.845+0000 7f17c833a800 20 remove_watcher() i=3 2023-12-03T11:36:37.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.846+0000 7f17c833a800 20 remove_watcher() i=4 2023-12-03T11:36:37.848 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.847+0000 7f17c833a800 20 remove_watcher() i=5 2023-12-03T11:36:37.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.848+0000 7f17c833a800 20 remove_watcher() i=6 2023-12-03T11:36:37.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:37.849+0000 7f17c833a800 20 remove_watcher() i=7 2023-12-03T11:36:37.855 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:37.855 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:37.855 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-set-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:36:37.855 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:37.942 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with EIO injected at block_writes 2023-12-03T11:36:37.943 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.157 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:36:38.184 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:38.184 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:38.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.198+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 realm 2023-12-03T11:36:38.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.200+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.201+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.201+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.202+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:38.203 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.202+0000 7f101cd1e800 20 searching for the correct realm 2023-12-03T11:36:38.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.212+0000 7f101b026640 0 -- 172.21.15.12:0/4131659220 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56301d2c4d00 msgr2=0x56301d2c71c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.215 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.213+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 zone default found 2023-12-03T11:36:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 4 Realm: () 2023-12-03T11:36:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:38.216 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 zonegroup default 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.214+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.215+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.215+0000 7f101cd1e800 20 started sync module instance, tier type = 2023-12-03T11:36:38.217 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.215+0000 7f101cd1e800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:38.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.221+0000 7f101b026640 0 -- 172.21.15.12:0/4131659220 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f0ff4079db0 msgr2=0x7f0ff407c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=0 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=1 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=2 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=3 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=4 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=5 2023-12-03T11:36:38.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.224+0000 7f101cd1e800 20 add_watcher() i=6 2023-12-03T11:36:38.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.225+0000 7f101cd1e800 20 add_watcher() i=7 2023-12-03T11:36:38.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.225+0000 7f101cd1e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:38.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.226+0000 7f0f5bfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.226+0000 7f101cd1e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:38.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.226+0000 7f101cd1e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:38.233 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.232+0000 7f101cd1e800 5 note: GC not initialized 2023-12-03T11:36:38.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f0f4affd640 20 reqs_thread_entry: start 2023-12-03T11:36:38.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f101cd1e800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:38.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f101cd1e800 20 Filter name: none 2023-12-03T11:36:38.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f0f48ff9640 20 reqs_thread_entry: start 2023-12-03T11:36:38.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.235 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.233+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.234+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.234+0000 7f101cd1e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.234+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.235+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.235+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.235+0000 7f101cd1e800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.235+0000 7f101cd1e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.235+0000 7f101cd1e800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.236+0000 7f101cd1e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.237 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.237+0000 7f0f5bfff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.239+0000 7f101cd1e800 20 remove_watcher() i=0 2023-12-03T11:36:38.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.239+0000 7f101cd1e800 2 removed watcher, disabling cache 2023-12-03T11:36:38.241 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.240+0000 7f101cd1e800 20 remove_watcher() i=1 2023-12-03T11:36:38.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.241+0000 7f101cd1e800 20 remove_watcher() i=2 2023-12-03T11:36:38.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.242+0000 7f101cd1e800 20 remove_watcher() i=3 2023-12-03T11:36:38.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.243+0000 7f101cd1e800 20 remove_watcher() i=4 2023-12-03T11:36:38.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.243+0000 7f101cd1e800 20 remove_watcher() i=5 2023-12-03T11:36:38.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.244+0000 7f101cd1e800 20 remove_watcher() i=6 2023-12-03T11:36:38.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.245+0000 7f101cd1e800 20 remove_watcher() i=7 2023-12-03T11:36:38.251 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.251 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:38.251 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:38.251 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.253 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at block_writes 2023-12-03T11:36:38.286 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:38.286 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:38.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.299+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 realm 2023-12-03T11:36:38.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.301+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.302+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.303 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.302+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.303+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:38.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.303+0000 7fb49e32f800 20 searching for the correct realm 2023-12-03T11:36:38.312 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.311+0000 7fb49a869640 0 -- 172.21.15.12:0/126425290 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x561e22364630 msgr2=0x561e22366af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.313+0000 7fb49e32f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.313+0000 7fb49e32f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.313+0000 7fb49e32f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.313+0000 7fb49e32f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.313+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.316 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 zone default found 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 4 Realm: () 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 zonegroup default 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.314+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.317 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 started sync module instance, tier type = 2023-12-03T11:36:38.318 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.315+0000 7fb49e32f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:38.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.319+0000 7fb499867640 0 -- 172.21.15.12:0/126425290 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x561e2235fd80 msgr2=0x561e22362170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=0 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=1 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=2 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=3 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=4 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=5 2023-12-03T11:36:38.325 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.324+0000 7fb49e32f800 20 add_watcher() i=6 2023-12-03T11:36:38.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.325+0000 7fb49e32f800 20 add_watcher() i=7 2023-12-03T11:36:38.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.325+0000 7fb49e32f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:38.327 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.326+0000 7fb45ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.327+0000 7fb49e32f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:38.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.327+0000 7fb49e32f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:38.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb49e32f800 5 note: GC not initialized 2023-12-03T11:36:38.334 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb3ceffd640 20 reqs_thread_entry: start 2023-12-03T11:36:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb49e32f800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb49e32f800 20 Filter name: none 2023-12-03T11:36:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb3ccff9640 20 reqs_thread_entry: start 2023-12-03T11:36:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb49e32f800 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2023-12-03T11:36:38.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.333+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 : miss 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.334+0000 7fb49e32f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 2023-12-03T11:36:38.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.338+0000 7fb49e32f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.338+0000 7fb49e32f800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.338+0000 7fb49e32f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.338+0000 7fb49e32f800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21, ofs0, ns] 2023-12-03T11:36:38.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb488ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725921 cookie 94687423007968 notifier 6636 bl.length()=877 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb488ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb488ff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb488ff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_sources= new_sources= 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential sources added=[] removed=[] 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_dests= new_dests= 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential dests added=[] removed=[] 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 -1 Injecting error=-5 at location=block_writes 2023-12-03T11:36:38.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.339+0000 7fb49e32f800 0 ERROR: init_reshard failed to pause writes to the current index: (5) Input/output error 2023-12-03T11:36:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.341+0000 7fb49e32f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:38.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.341+0000 7fb49e32f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 2023-12-03T11:36:38.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21, ofs0, ns] 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb47bfff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725922 cookie 94687423007968 notifier 6636 bl.length()=845 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb47bfff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb47bfff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb47bfff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_sources= new_sources= 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential sources added=[] removed=[] 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_dests= new_dests= 2023-12-03T11:36:38.344 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.342+0000 7fb49e32f800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential dests added=[] removed=[] 2023-12-03T11:36:38.345 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.344+0000 7fb45ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.346+0000 7fb49e32f800 20 remove_watcher() i=0 2023-12-03T11:36:38.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.346+0000 7fb49e32f800 2 removed watcher, disabling cache 2023-12-03T11:36:38.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.347+0000 7fb49e32f800 20 remove_watcher() i=1 2023-12-03T11:36:38.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.348+0000 7fb49e32f800 20 remove_watcher() i=2 2023-12-03T11:36:38.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.349+0000 7fb49e32f800 20 remove_watcher() i=3 2023-12-03T11:36:38.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.350+0000 7fb49e32f800 20 remove_watcher() i=4 2023-12-03T11:36:38.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.351+0000 7fb49e32f800 20 remove_watcher() i=5 2023-12-03T11:36:38.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.352+0000 7fb49e32f800 20 remove_watcher() i=6 2023-12-03T11:36:38.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.353+0000 7fb49e32f800 20 remove_watcher() i=7 2023-12-03T11:36:38.360 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:36:38.388 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:38.388 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:38.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.596+0000 7f2c054a3640 0 --2- 172.21.15.12:0/1393500751 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x5580c09d8220 0x5580c0a70dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:36:38.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.603+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.604+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 realm 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.605+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.606+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.606+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.607+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:38.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.607+0000 7f2c0973d800 20 searching for the correct realm 2023-12-03T11:36:38.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.614+0000 7f2c054a3640 0 -- 172.21.15.12:0/319168603 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f2bfc0853f0 msgr2=0x7f2bfc085870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.617+0000 7f2c0973d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.617+0000 7f2c0973d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.617+0000 7f2c0973d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.617+0000 7f2c0973d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.617+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 zone default found 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 4 Realm: () 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 zonegroup default 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.618+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 started sync module instance, tier type = 2023-12-03T11:36:38.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.619+0000 7f2c0973d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:38.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.620+0000 7f2c05ca4640 0 -- 172.21.15.12:0/319168603 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f2bfc056070 msgr2=0x7f2bfc056450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=0 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=1 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=2 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=3 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=4 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=5 2023-12-03T11:36:38.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.628+0000 7f2c0973d800 20 add_watcher() i=6 2023-12-03T11:36:38.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.629+0000 7f2c0973d800 20 add_watcher() i=7 2023-12-03T11:36:38.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.629+0000 7f2c0973d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:38.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.630+0000 7f2bcafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.631+0000 7f2c0973d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:38.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.631+0000 7f2c0973d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.636+0000 7f2c0973d800 5 note: GC not initialized 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2b39ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2c0973d800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2c0973d800 20 Filter name: none 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2b2b7fe640 20 reqs_thread_entry: start 2023-12-03T11:36:38.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.637+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.639 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.638+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.639+0000 7f2c0973d800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.639+0000 7f2c0973d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.639+0000 7f2c0973d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.639+0000 7f2c0973d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.640+0000 7f2bcafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.642+0000 7f2c0973d800 20 remove_watcher() i=0 2023-12-03T11:36:38.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.642+0000 7f2c0973d800 2 removed watcher, disabling cache 2023-12-03T11:36:38.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.644+0000 7f2c0973d800 20 remove_watcher() i=1 2023-12-03T11:36:38.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.645+0000 7f2c0973d800 20 remove_watcher() i=2 2023-12-03T11:36:38.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.646+0000 7f2c0973d800 20 remove_watcher() i=3 2023-12-03T11:36:38.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.647+0000 7f2c0973d800 20 remove_watcher() i=4 2023-12-03T11:36:38.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.648+0000 7f2c0973d800 20 remove_watcher() i=5 2023-12-03T11:36:38.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.648+0000 7f2c0973d800 20 remove_watcher() i=6 2023-12-03T11:36:38.650 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.649+0000 7f2c0973d800 20 remove_watcher() i=7 2023-12-03T11:36:38.655 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.655 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:38.655 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:38.656 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.708 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:36:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:38.739 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:38.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.752+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 realm 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.754+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.755+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.755+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.756+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:38.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.756+0000 7fc79312d800 20 searching for the correct realm 2023-12-03T11:36:38.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.765+0000 7fc78edd7640 0 -- 172.21.15.12:0/1913387625 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7fc778099a40 msgr2=0x7fc778098740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.767+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 zone default found 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 4 Realm: () 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 zonegroup default 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.768+0000 7fc79312d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.769+0000 7fc79312d800 20 started sync module instance, tier type = 2023-12-03T11:36:38.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.769+0000 7fc79312d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:38.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.774+0000 7fc78edd7640 0 -- 172.21.15.12:0/1913387625 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7fc768079d80 msgr2=0x7fc76807c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=0 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=1 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=2 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=3 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=4 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=5 2023-12-03T11:36:38.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.778+0000 7fc79312d800 20 add_watcher() i=6 2023-12-03T11:36:38.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.779+0000 7fc79312d800 20 add_watcher() i=7 2023-12-03T11:36:38.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.779+0000 7fc79312d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.780+0000 7fc6cffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.780+0000 7fc79312d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:38.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.780+0000 7fc79312d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.786+0000 7fc79312d800 5 note: GC not initialized 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc6beffd640 20 reqs_thread_entry: start 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc79312d800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc79312d800 20 Filter name: none 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc6bcff9640 20 reqs_thread_entry: start 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc79312d800 10 cache get: name=default.rgw.meta+root+error-at-block-writes : miss 2023-12-03T11:36:38.788 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.787+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 cache put: name=default.rgw.meta+root+error-at-block-writes info.flags=0x11 2023-12-03T11:36:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 adding default.rgw.meta+root+error-at-block-writes to cache LRU end 2023-12-03T11:36:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 : miss 2023-12-03T11:36:38.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 adding default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.788+0000 7fc79312d800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 2023-12-03T11:36:38.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21, ofs0, ns] 2023-12-03T11:36:38.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc7857fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725923 cookie 94536034879056 notifier 6344 bl.length()=877 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc7857fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc7857fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc7857fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_sources= new_sources= 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential sources added=[] removed=[] 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_dests= new_dests= 2023-12-03T11:36:38.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.793+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential dests added=[] removed=[] 2023-12-03T11:36:38.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.794+0000 7fc79312d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21.1.0 2023-12-03T11:36:38.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.794+0000 7fc79312d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21.1.0 2023-12-03T11:36:38.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.794+0000 7fc79312d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21.1.1 2023-12-03T11:36:38.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.794+0000 7fc79312d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21.1.1 2023-12-03T11:36:38.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.794+0000 7fc79312d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21.0 2023-12-03T11:36:38.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.797+0000 7fc79312d800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 : hit (requested=0x17, cached=0x17) 2023-12-03T11:36:38.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.797+0000 7fc79312d800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 2023-12-03T11:36:38.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc79312d800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc79312d800 10 moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc79312d800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc79312d800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21, ofs0, ns] 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc784ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725924 cookie 94536034879056 notifier 6344 bl.length()=845 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc784ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 info.flags=0x17 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc784ff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 to cache LRU end 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.798+0000 7fc784ff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.799+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_sources= new_sources= 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.799+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential sources added=[] removed=[] 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.799+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): orig_dests= new_dests= 2023-12-03T11:36:38.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.799+0000 7fc79312d800 20 handle_bi_update(): bucket=:error-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21]): potential dests added=[] removed=[] 2023-12-03T11:36:38.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.800+0000 7fc79312d800 1 execute INFO: reshard of bucket "error-at-block-writes" completed successfully 2023-12-03T11:36:38.802 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.801+0000 7fc6cffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.803+0000 7fc79312d800 20 remove_watcher() i=0 2023-12-03T11:36:38.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.803+0000 7fc79312d800 2 removed watcher, disabling cache 2023-12-03T11:36:38.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.804+0000 7fc79312d800 20 remove_watcher() i=1 2023-12-03T11:36:38.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.805+0000 7fc79312d800 20 remove_watcher() i=2 2023-12-03T11:36:38.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.806+0000 7fc79312d800 20 remove_watcher() i=3 2023-12-03T11:36:38.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.808+0000 7fc79312d800 20 remove_watcher() i=4 2023-12-03T11:36:38.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.808+0000 7fc79312d800 20 remove_watcher() i=5 2023-12-03T11:36:38.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.809+0000 7fc79312d800 20 remove_watcher() i=6 2023-12-03T11:36:38.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.810+0000 7fc79312d800 20 remove_watcher() i=7 2023-12-03T11:36:38.818 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-block-writes 2023-12-03T11:36:38.845 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:38.845 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:38.860 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.859+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 realm 2023-12-03T11:36:38.862 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.861+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.862+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.862+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.863+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:38.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.863+0000 7f641eab8800 20 searching for the correct realm 2023-12-03T11:36:38.871 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.870+0000 7f641bdb7640 0 -- 172.21.15.12:0/3927989907 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x56216c10a6f0 msgr2=0x56216c10cae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.873+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 zone default found 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 4 Realm: () 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:38.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 zonegroup default 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.874+0000 7f641cdb9640 0 -- 172.21.15.12:0/3927989907 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56216c10f050 msgr2=0x56216c1114e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:38.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.875+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.875+0000 7f641eab8800 20 started sync module instance, tier type = 2023-12-03T11:36:38.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.875+0000 7f641eab8800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:38.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=0 2023-12-03T11:36:38.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=1 2023-12-03T11:36:38.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=2 2023-12-03T11:36:38.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=3 2023-12-03T11:36:38.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=4 2023-12-03T11:36:38.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=5 2023-12-03T11:36:38.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.885+0000 7f641eab8800 20 add_watcher() i=6 2023-12-03T11:36:38.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.886+0000 7f641eab8800 20 add_watcher() i=7 2023-12-03T11:36:38.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.886+0000 7f641eab8800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:38.888 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.887+0000 7f63e27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.888+0000 7f641eab8800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:38.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.888+0000 7f641eab8800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:38.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.893+0000 7f641eab8800 5 note: GC not initialized 2023-12-03T11:36:38.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f63517fa640 20 reqs_thread_entry: start 2023-12-03T11:36:38.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f641eab8800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:38.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f641eab8800 20 Filter name: none 2023-12-03T11:36:38.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f634affd640 20 reqs_thread_entry: start 2023-12-03T11:36:38.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.894+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.895+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.895+0000 7f641eab8800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.895+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.896+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:38.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.896+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.896+0000 7f641eab8800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:38.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.896+0000 7f641eab8800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:38.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.896+0000 7f641eab8800 20 rados->read ofs=0 len=0 2023-12-03T11:36:38.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.897+0000 7f641eab8800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:38.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.897+0000 7f63e27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:38.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.899+0000 7f641eab8800 20 remove_watcher() i=0 2023-12-03T11:36:38.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.899+0000 7f641eab8800 2 removed watcher, disabling cache 2023-12-03T11:36:38.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.901+0000 7f641eab8800 20 remove_watcher() i=1 2023-12-03T11:36:38.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.902+0000 7f641eab8800 20 remove_watcher() i=2 2023-12-03T11:36:38.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.903+0000 7f641eab8800 20 remove_watcher() i=3 2023-12-03T11:36:38.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.904+0000 7f641eab8800 20 remove_watcher() i=4 2023-12-03T11:36:38.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.905+0000 7f641eab8800 20 remove_watcher() i=5 2023-12-03T11:36:38.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.906+0000 7f641eab8800 20 remove_watcher() i=6 2023-12-03T11:36:38.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:38.907+0000 7f641eab8800 20 remove_watcher() i=7 2023-12-03T11:36:38.913 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.913 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:38.913 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:36:38.913 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:38.995 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with abort at block_writes 2023-12-03T11:36:38.995 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.211 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:36:39.238 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:39.238 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:39.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.252+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 realm 2023-12-03T11:36:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.254+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.255+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.255+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.256+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.256+0000 7efe34c12800 20 searching for the correct realm 2023-12-03T11:36:39.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.263+0000 7efe30b2b640 0 -- 172.21.15.12:0/3032233704 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7efe24059ba0 msgr2=0x7efe24061b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.269 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 zone default found 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 4 Realm: () 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 zonegroup default 2023-12-03T11:36:39.270 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.267+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe3032a640 0 -- 172.21.15.12:0/3032233704 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7efe2405b990 msgr2=0x7efe2408a210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 started sync module instance, tier type = 2023-12-03T11:36:39.271 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.268+0000 7efe34c12800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=0 2023-12-03T11:36:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=1 2023-12-03T11:36:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=2 2023-12-03T11:36:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=3 2023-12-03T11:36:39.278 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=4 2023-12-03T11:36:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=5 2023-12-03T11:36:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.277+0000 7efe34c12800 20 add_watcher() i=6 2023-12-03T11:36:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.278+0000 7efe34c12800 20 add_watcher() i=7 2023-12-03T11:36:39.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.278+0000 7efe34c12800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.280+0000 7efdf67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.280+0000 7efe34c12800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.280+0000 7efe34c12800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.287 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efe34c12800 5 note: GC not initialized 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efd657fa640 20 reqs_thread_entry: start 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efe34c12800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efe34c12800 20 Filter name: none 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efd5effd640 20 reqs_thread_entry: start 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.286+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.287+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.288 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.287+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.288+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:39.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.288+0000 7efe34c12800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:39.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.288+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.289 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.288+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.288+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.290 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.289+0000 7efe34c12800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:39.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.289+0000 7efe34c12800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:39.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.289+0000 7efe34c12800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.289+0000 7efe34c12800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.291 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.290+0000 7efdf67e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.292+0000 7efe34c12800 20 remove_watcher() i=0 2023-12-03T11:36:39.293 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.292+0000 7efe34c12800 2 removed watcher, disabling cache 2023-12-03T11:36:39.294 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.293+0000 7efe34c12800 20 remove_watcher() i=1 2023-12-03T11:36:39.295 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.294+0000 7efe34c12800 20 remove_watcher() i=2 2023-12-03T11:36:39.296 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.295+0000 7efe34c12800 20 remove_watcher() i=3 2023-12-03T11:36:39.297 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.296+0000 7efe34c12800 20 remove_watcher() i=4 2023-12-03T11:36:39.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.297+0000 7efe34c12800 20 remove_watcher() i=5 2023-12-03T11:36:39.299 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.298+0000 7efe34c12800 20 remove_watcher() i=6 2023-12-03T11:36:39.300 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.299+0000 7efe34c12800 20 remove_watcher() i=7 2023-12-03T11:36:39.305 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.305 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:39.305 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:39.305 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.307 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at block_writes 2023-12-03T11:36:39.339 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:39.339 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:39.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.353+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.354+0000 7fa610f24800 20 realm 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.355+0000 7fa610f24800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.357+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.357+0000 7fa610f24800 20 searching for the correct realm 2023-12-03T11:36:39.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.366+0000 7fa60cc5a640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x555cc44d48d0 msgr2=0x555cc44d6ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.367+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 zone default found 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 4 Realm: () 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 zonegroup default 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.369+0000 7fa610f24800 20 started sync module instance, tier type = 2023-12-03T11:36:39.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.369+0000 7fa610f24800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.377 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.376+0000 7fa60d45b640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fa600017070 msgr2=0x7fa600077990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=0 2023-12-03T11:36:39.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=1 2023-12-03T11:36:39.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=2 2023-12-03T11:36:39.379 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=3 2023-12-03T11:36:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=4 2023-12-03T11:36:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=5 2023-12-03T11:36:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=6 2023-12-03T11:36:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.379+0000 7fa610f24800 20 add_watcher() i=7 2023-12-03T11:36:39.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.379+0000 7fa610f24800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.381+0000 7fa5d2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.387+0000 7fa610f24800 5 note: GC not initialized 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.387+0000 7fa541ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.387+0000 7fa610f24800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.387+0000 7fa610f24800 20 Filter name: none 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.387+0000 7fa5337fe640 20 reqs_thread_entry: start 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:36:39.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : miss 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.390 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:39.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.389+0000 7fa610f24800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:36:39.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22, ofs0, ns] 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725969 cookie 93856918801520 notifier 6744 bl.length()=877 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_sources= new_sources= 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential sources added=[] removed=[] 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_dests= new_dests= 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fa610f24800 time 2023-12-03T11:36:39.394832+0000 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:39.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential dests added=[] removed=[] 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa613b6eef8] 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.410 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7fa610f24800 thread_name:radosgw-admin 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.394+0000 7fa610f24800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fa610f24800 time 2023-12-03T11:36:39.394832+0000 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa613b6eef8] 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:36:39.411 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa611454db0] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa6114a154c] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa613b6ef52] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.412 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.395+0000 7fa610f24800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7fa610f24800 thread_name:radosgw-admin 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa611454db0] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa6114a154c] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa613b6ef52] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.413 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:36:39.347+0000 7fa60b457640 4 set_mon_vals no callback set 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:36:39.347+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: get_monmap_and_config 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.414 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:36:39.347+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.415 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:36:39.348+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.416 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:36:39.348+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.417 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:36:39.348+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:36:39.349+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:36:39.349+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x7ffe35906f08 EEXIST 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc44b2b30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc44b2f10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc44aaa20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.531830s 2023-12-03T11:36:39.418 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc44b2b30 auth_method 0 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc44b2f10 auth_method 0 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc44aaa20 auth_method 0 2023-12-03T11:36:39.419 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6741 payload 274 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:36:39.349+0000 7fa60d45b640 1 monclient: found mon.noname-a 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5f8069820 auth_method 0 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: dump: 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.420 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: dump: 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.421 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 monclient: authenticate success, global_id 6741 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:36:39.350+0000 7fa610f24800 4 set_mon_vals no callback set 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: shutdown 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.422 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.423 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:36:39.350+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 librados: starting msgr at 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 librados: starting objecter 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command objecter_requests cmddesc objecter_requests hook 0x555cc4460c20 EEXIST 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:36:39.351+0000 7fa610f24800 1 librados: setting wanted keys 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:36:39.351+0000 7fa610f24800 1 librados: calling monclient init 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.424 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:36:39.351+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:36:39.351+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.425 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x555cc44823a0 EEXIST 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc44aaa20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc44b2b30 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc44b0670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.534139s 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc44b0670 auth_method 0 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.426 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc44aaa20 auth_method 0 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc44b2b30 auth_method 0 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.427 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6744 payload 824 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:36:39.351+0000 7fa60d45b640 1 monclient: found mon.noname-a 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: dump: 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.428 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: dump: 2023-12-03T11:36:39.429 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:36:39.352+0000 7fa610f24800 5 monclient: authenticate success, global_id 6744 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:36:39.352+0000 7fa5f37fe640 4 set_mon_vals no callback set 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.430 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:36:39.352+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:36:39.353+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:36:39.353+0000 7fa5f1ffb640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:36:39.354+0000 7fa5f1ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:36:39.354+0000 7fa5f1ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:36:39.354+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5e8069850 auth_method 0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:36:39.354+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc44d0a10 auth_method 0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 realm 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.431 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:36:39.355+0000 7fa610f24800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:36:39.355+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x555cc44d48d0 auth_method 0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:36:39.356+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc44d8110 auth_method 0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:36:39.357+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:36:39.357+0000 7fa610f24800 20 searching for the correct realm 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:36:39.357+0000 7fa60d45b640 10 monclient: get_auth_request con 0x555cc44dc9a0 auth_method 0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:36:39.358+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa600092040 auth_method 0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:36:39.358+0000 7fa60c459640 10 monclient: get_auth_request con 0x7fa5fc027070 auth_method 0 2023-12-03T11:36:39.432 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:36:39.360+0000 7fa60d45b640 10 monclient: get_auth_request con 0x7fa600017070 auth_method 0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:36:39.362+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa600059050 auth_method 0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:36:39.366+0000 7fa60cc5a640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x555cc44d48d0 msgr2=0x555cc44d6ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 1 client.6744.objecter ms_handle_reset 0x555cc44d48d0 session 0x555cc4242320 osd.5 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 10 monclient: _renew_subs 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:36:39.366+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5e8079db0 auth_method 0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.433 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 zone default found 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 Realm: () 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:36:39.368+0000 7fa610f24800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 zonegroup default 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.434 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 started sync module instance, tier type = 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:36:39.376+0000 7fa60d45b640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fa600017070 msgr2=0x7fa600077990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:36:39.376+0000 7fa5f1ffb640 1 client.6744.objecter ms_handle_reset 0x7fa600017070 session 0x7fa600088e60 osd.6 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:36:39.376+0000 7fa60d45b640 10 monclient: get_auth_request con 0x555cc44d48d0 auth_method 0 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=0 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=1 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=2 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=3 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=4 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=5 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=6 2023-12-03T11:36:39.435 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:36:39.379+0000 7fa610f24800 20 add_watcher() i=7 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:36:39.379+0000 7fa610f24800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:36:39.381+0000 7fa5d2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.436 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:36:39.381+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache list hook 0x555cc446a210 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache inspect hook 0x555cc446a210 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache erase hook 0x555cc446a210 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache zap hook 0x555cc446a210 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: get_monmap_and_config 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:36:39.381+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.437 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.438 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:36:39.381+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.439 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:36:39.382+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.440 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:36:39.382+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x7ffe35907488 EEXIST 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc451ad70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc4515960 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc45227b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.565122s 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc45227b0 auth_method 0 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.441 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc451ad70 auth_method 0 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc4515960 auth_method 0 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.442 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6404 payload 274 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:36:39.382+0000 7fa60d45b640 1 monclient: found mon.noname-c 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: dump: 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.443 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:36:39.383+0000 7fa610f24800 5 monclient: authenticate success, global_id 6404 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:36:39.383+0000 7fa610f24800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:36:39.383+0000 7fa610f24800 4 set_mon_vals no callback set 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:39.444 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: dump: 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:36:39.383+0000 7fa610f24800 10 monclient: shutdown 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.445 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.446 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: starting msgr at 2023-12-03T11:36:39.447 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: starting objecter 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command objecter_requests cmddesc objecter_requests hook 0x555cc4515f20 EEXIST 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: setting wanted keys 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: calling monclient init 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.448 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x555cc451bb30 EEXIST 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc4515960 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc451ad70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc45227b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.449 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.567621s 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc4515960 auth_method 0 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc451ad70 auth_method 0 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc45227b0 auth_method 0 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.450 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6407 payload 824 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:36:39.385+0000 7fa60d45b640 1 monclient: found mon.noname-c 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: dump: 2023-12-03T11:36:39.451 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:39.452 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: dump: 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:36:39.385+0000 7fa610f24800 5 monclient: authenticate success, global_id 6407 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:36:39.385+0000 7fa5d27e4640 4 set_mon_vals no callback set 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.453 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:36:39.385+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:36:39.385+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cr dump hook 0x555cc4521848 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:36:39.387+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa548069820 auth_method 0 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 note: GC not initialized 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:36:39.387+0000 7fa541ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace show hook 0x7fa604004720 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace history hook 0x7fa604004720 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace active hook 0x7fa604004720 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace active_short hook 0x7fa604004720 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:36:39.387+0000 7fa610f24800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:36:39.387+0000 7fa610f24800 20 Filter name: none 2023-12-03T11:36:39.454 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:36:39.387+0000 7fa5337fe640 20 reqs_thread_entry: start 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : miss 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:36:39.389+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:36:39.389+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc4568750 auth_method 0 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.455 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22, ofs0, ns] 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725969 cookie 93856918801520 notifier 6744 bl.length()=877 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_sources= new_sources= 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential sources added=[] removed=[] 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_dests= new_dests= 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential dests added=[] removed=[] 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:36:39.394+0000 7fa610f24800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fa610f24800 time 2023-12-03T11:36:39.394832+0000 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.456 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa613b6eef8] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:36:39.395+0000 7fa610f24800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7fa610f24800 thread_name:radosgw-admin 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa611454db0] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa6114a154c] 2023-12-03T11:36:39.457 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa613b6ef52] 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.458 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:36:39.459 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:36:39.460 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:36:39.461 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:36:39.462 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:36:39.463 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:36:39.464 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5337fe640 / http_manager 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa541ffb640 / http_manager 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d0fe1640 / ms_dispatch 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d27e4640 / io_context_pool 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d2fe5640 / rgw_dt_lg_renew 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5f1ffb640 / ms_dispatch 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5f37fe640 / io_context_pool 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60a455640 / ms_dispatch 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60b457640 / io_context_pool 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60c459640 / msgr-worker-2 2023-12-03T11:36:39.465 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60cc5a640 / msgr-worker-1 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60d45b640 / msgr-worker-0 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa610f24800 / radosgw-admin 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/log/ceph/ceph-client.admin.102341.log 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:36:39.347+0000 7fa60b457640 4 set_mon_vals no callback set 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.466 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:36:39.347+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: get_monmap_and_config 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:36:39.347+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:36:39.347+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.467 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding auth protocol: cephx 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.468 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: crc 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:36:39.347+0000 7fa610f24800 5 AuthRegistry(0x555cc4483f30) adding con mode: secure 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:36:39.348+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:36:39.348+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding auth protocol: cephx 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.469 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: crc 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:36:39.348+0000 7fa610f24800 5 AuthRegistry(0x7ffe35906dc0) adding con mode: secure 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:36:39.348+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:36:39.349+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:36:39.349+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x7ffe35906f08 EEXIST 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient: _add_conns ranks=[0,2,1] 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc44b2b30 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc44b2f10 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc44aaa20 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.470 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:36:39.349+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.531830s 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc44b2b30 auth_method 0 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc44b2f10 auth_method 0 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.471 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc44aaa20 auth_method 0 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:36:39.349+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6741 payload 274 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:36:39.349+0000 7fa60d45b640 1 monclient: found mon.noname-a 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:36:39.349+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:36:39.349+0000 7fa60a455640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:36:39.349+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5f8069820 auth_method 0 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.472 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: dump: 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.473 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:36:39.349+0000 7fa5f37fe640 10 monclient: dump: 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 monclient: authenticate success, global_id 6741 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:39.474 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:36:39.350+0000 7fa610f24800 4 set_mon_vals no callback set 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: shutdown 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:36:39.350+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.475 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding auth protocol: cephx 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: crc 2023-12-03T11:36:39.476 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 AuthRegistry(0x555cc4483e00) adding con mode: secure 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:36:39.350+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 librados: starting msgr at 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:36:39.350+0000 7fa610f24800 1 librados: starting objecter 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:36:39.350+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command objecter_requests cmddesc objecter_requests hook 0x555cc4460c20 EEXIST 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:36:39.351+0000 7fa610f24800 1 librados: setting wanted keys 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:36:39.351+0000 7fa610f24800 1 librados: calling monclient init 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding auth protocol: cephx 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.477 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: crc 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 AuthRegistry(0x555cc4482258) adding con mode: secure 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:36:39.351+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:36:39.351+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:36:39.351+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x555cc44823a0 EEXIST 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.478 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc44aaa20 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc44b2b30 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc44b0670 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:36:39.351+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.534139s 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc44b0670 auth_method 0 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc44aaa20 auth_method 0 2023-12-03T11:36:39.479 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc44b2b30 auth_method 0 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:36:39.351+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.480 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:36:39.351+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6744 payload 824 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:36:39.351+0000 7fa60d45b640 1 monclient: found mon.noname-a 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:36:39.351+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: dump: 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.481 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:36:39.352+0000 7fa5f1ffb640 10 monclient: dump: 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.482 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:36:39.352+0000 7fa610f24800 5 monclient: authenticate success, global_id 6744 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:36:39.352+0000 7fa5f37fe640 4 set_mon_vals no callback set 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:36:39.352+0000 7fa610f24800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:36:39.352+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:36:39.353+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.483 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:36:39.353+0000 7fa5f1ffb640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:36:39.354+0000 7fa5f1ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:36:39.354+0000 7fa5f1ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:36:39.354+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5e8069850 auth_method 0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:36:39.354+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc44d0a10 auth_method 0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 realm 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:36:39.354+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:36:39.355+0000 7fa610f24800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:36:39.355+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.484 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:36:39.355+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x555cc44d48d0 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:36:39.356+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:36:39.356+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc44d8110 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:36:39.357+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:36:39.357+0000 7fa610f24800 20 searching for the correct realm 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:36:39.357+0000 7fa60d45b640 10 monclient: get_auth_request con 0x555cc44dc9a0 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:36:39.358+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa600092040 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:36:39.358+0000 7fa60c459640 10 monclient: get_auth_request con 0x7fa5fc027070 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:36:39.360+0000 7fa60d45b640 10 monclient: get_auth_request con 0x7fa600017070 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:36:39.362+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa600059050 auth_method 0 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:36:39.366+0000 7fa60cc5a640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x555cc44d48d0 msgr2=0x555cc44d6ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 1 client.6744.objecter ms_handle_reset 0x555cc44d48d0 session 0x555cc4242320 osd.5 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 10 monclient: _renew_subs 2023-12-03T11:36:39.485 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:36:39.366+0000 7fa5f1ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:36:39.366+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa5e8079db0 auth_method 0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:36:39.367+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 zone default found 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 Realm: () 2023-12-03T11:36:39.486 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:36:39.368+0000 7fa610f24800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:36:39.368+0000 7fa610f24800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 zonegroup default 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:36:39.368+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 started sync module instance, tier type = 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:36:39.369+0000 7fa610f24800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:36:39.376+0000 7fa60d45b640 0 -- 172.21.15.12:0/3831349113 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fa600017070 msgr2=0x7fa600077990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.487 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:36:39.376+0000 7fa5f1ffb640 1 client.6744.objecter ms_handle_reset 0x7fa600017070 session 0x7fa600088e60 osd.6 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:36:39.376+0000 7fa60d45b640 10 monclient: get_auth_request con 0x555cc44d48d0 auth_method 0 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=0 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=1 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=2 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=3 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=4 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=5 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:36:39.378+0000 7fa610f24800 20 add_watcher() i=6 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:36:39.379+0000 7fa610f24800 20 add_watcher() i=7 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:36:39.379+0000 7fa610f24800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.488 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding auth protocol: cephx 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:36:39.381+0000 7fa5d2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.489 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: crc 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x7ffe359077f0) adding con mode: secure 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:36:39.381+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:36:39.381+0000 7fa610f24800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache list hook 0x555cc446a210 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache inspect hook 0x555cc446a210 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache erase hook 0x555cc446a210 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cache zap hook 0x555cc446a210 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: get_monmap_and_config 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:36:39.381+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.490 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding auth protocol: cephx 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.491 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: crc 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:36:39.381+0000 7fa610f24800 5 AuthRegistry(0x555cc45184d0) adding con mode: secure 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:36:39.381+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:36:39.381+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.492 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding auth protocol: cephx 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: crc 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 AuthRegistry(0x7ffe35907340) adding con mode: secure 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:36:39.382+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:36:39.382+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.493 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:36:39.382+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x7ffe35907488 EEXIST 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,1,0] 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc451ad70 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc4515960 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc45227b0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:36:39.382+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.565122s 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc45227b0 auth_method 0 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.494 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc451ad70 auth_method 0 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc4515960 auth_method 0 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:36:39.382+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.495 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:36:39.382+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6404 payload 274 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:36:39.382+0000 7fa60d45b640 1 monclient: found mon.noname-c 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:36:39.382+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: dump: 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.496 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:36:39.383+0000 7fa610f24800 5 monclient: authenticate success, global_id 6404 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:36:39.383+0000 7fa610f24800 10 monclient: get_monmap_and_config success 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:36:39.383+0000 7fa610f24800 4 set_mon_vals no callback set 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.497 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:36:39.383+0000 7fa5d27e4640 10 monclient: dump: 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:36:39.383+0000 7fa610f24800 10 monclient: shutdown 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: build_initial_monmap 2023-12-03T11:36:39.498 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 build_initial for_mkfs: 0 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: monmap: 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding auth protocol: cephx 2023-12-03T11:36:39.499 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: crc 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc450dac0) adding con mode: secure 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: starting msgr at 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: starting objecter 2023-12-03T11:36:39.500 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command objecter_requests cmddesc objecter_requests hook 0x555cc4515f20 EEXIST 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: setting wanted keys 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:36:39.384+0000 7fa610f24800 1 librados: calling monclient init 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: init 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding auth protocol: cephx 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.501 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: crc 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 AuthRegistry(0x555cc451b9e8) adding con mode: secure 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:36:39.384+0000 7fa610f24800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:36:39.384+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command rotate-key cmddesc rotate-key hook 0x555cc451bb30 EEXIST 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: _reopen_session rank -1 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:36:39.384+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-c con 0x555cc4515960 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-a con 0x555cc451ad70 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): picked mon.noname-b con 0x555cc45227b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.502 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): start opening mon connection 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): _renew_subs 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient(hunting): authenticate will time out at 1609.567621s 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): get_auth_request con 0x555cc4515960 auth_method 0 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request con 0x555cc451ad70 auth_method 0 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): get_auth_request con 0x555cc45227b0 auth_method 0 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.503 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:36:39.385+0000 7fa60cc5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:36:39.385+0000 7fa60c459640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient(hunting): handle_auth_done global_id 6407 payload 824 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient: _finish_hunting 0 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:36:39.385+0000 7fa60d45b640 1 monclient: found mon.noname-c 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:36:39.385+0000 7fa60d45b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: dump: 2023-12-03T11:36:39.504 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: _finish_auth 0 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:36:39.385+0000 7fa5d0fe1640 10 monclient: dump: 2023-12-03T11:36:39.505 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:36:39.385+0000 7fa610f24800 5 monclient: authenticate success, global_id 6407 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:36:39.385+0000 7fa5d27e4640 4 set_mon_vals no callback set 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _renew_subs 2023-12-03T11:36:39.506 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:36:39.385+0000 7fa610f24800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:36:39.385+0000 7fa610f24800 1 librados: init done 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:36:39.385+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command cr dump hook 0x555cc4521848 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:36:39.387+0000 7fa5d0fe1640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:36:39.387+0000 7fa60cc5a640 10 monclient: get_auth_request con 0x7fa548069820 auth_method 0 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 note: GC not initialized 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:36:39.387+0000 7fa541ffb640 20 reqs_thread_entry: start 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace show hook 0x7fa604004720 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace history hook 0x7fa604004720 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace active hook 0x7fa604004720 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:36:39.387+0000 7fa610f24800 5 asok(0x555cc4279a10) register_command sync trace active_short hook 0x7fa604004720 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:36:39.387+0000 7fa610f24800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:36:39.387+0000 7fa610f24800 20 Filter name: none 2023-12-03T11:36:39.507 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:36:39.387+0000 7fa5337fe640 20 reqs_thread_entry: start 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:36:39.388+0000 7fa610f24800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : miss 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:36:39.388+0000 7fa610f24800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:36:39.389+0000 7fa610f24800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.508 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:36:39.389+0000 7fa610f24800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:36:39.389+0000 7fa60c459640 10 monclient: get_auth_request con 0x555cc4568750 auth_method 0 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:36:39.393+0000 7fa610f24800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22, ofs0, ns] 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725969 cookie 93856918801520 notifier 6744 bl.length()=877 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:36:39.393+0000 7fa5f37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_sources= new_sources= 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential sources added=[] removed=[] 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_dests= new_dests= 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:36:39.393+0000 7fa610f24800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential dests added=[] removed=[] 2023-12-03T11:36:39.509 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:36:39.394+0000 7fa610f24800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7fa610f24800 time 2023-12-03T11:36:39.394832+0000 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fa613b6eef8] 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.510 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:36:39.395+0000 7fa610f24800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7fa610f24800 thread_name:radosgw-admin 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7fa611454db0] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7fa6114a154c] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fa613b6ef52] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7fa613b6f0b6] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x555cc3376a71] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x2a4) [0x555cc3382ec4] 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:36:39.511 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7fa61143feb0] 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:36:39.512 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:36:39.513 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:36:39.514 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:36:39.515 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:36:39.516 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:36:39.517 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5337fe640 / http_manager 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa541ffb640 / http_manager 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d0fe1640 / ms_dispatch 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d27e4640 / io_context_pool 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5d2fe5640 / rgw_dt_lg_renew 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5f1ffb640 / ms_dispatch 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa5f37fe640 / io_context_pool 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60a455640 / ms_dispatch 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60b457640 / io_context_pool 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60c459640 / msgr-worker-2 2023-12-03T11:36:39.518 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60cc5a640 / msgr-worker-1 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa60d45b640 / msgr-worker-0 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr: 7fa610f24800 / radosgw-admin 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:36:39.396182Z_6857f2f0-2bca-4937-8ca1-49b459fe5301/log 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:36:39.519 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.442+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 realm 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.445+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.446+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.446+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.447+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.447+0000 7f2a287a9800 20 searching for the correct realm 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.455+0000 7f2a26aaa640 0 -- 172.21.15.12:0/2738755448 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x7f2a00079d80 msgr2=0x7f2a0007c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.457+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 zone default found 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 4 Realm: () 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 zonegroup default 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.458+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.459+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.459+0000 7f2a287a9800 20 started sync module instance, tier type = 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.459+0000 7f2a287a9800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.462+0000 7f2a25aa8640 0 -- 172.21.15.12:0/2738755448 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55c7d4aeb490 msgr2=0x55c7d4aed880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=0 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=1 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=2 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=3 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=4 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=5 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.467+0000 7f2a287a9800 20 add_watcher() i=6 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.468+0000 7f2a287a9800 20 add_watcher() i=7 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.468+0000 7f2a287a9800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.470+0000 7f29ebfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.470+0000 7f2a287a9800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.470+0000 7f2a287a9800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.476+0000 7f2a287a9800 5 note: GC not initialized 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.476+0000 7f295affd640 20 reqs_thread_entry: start 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2a287a9800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2a287a9800 20 Filter name: none 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2958ff9640 20 reqs_thread_entry: start 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.477+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.478+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.478+0000 7f2a287a9800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.478+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.478+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.478+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.479+0000 7f2a287a9800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.479+0000 7f2a287a9800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.479+0000 7f2a287a9800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.479+0000 7f2a287a9800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.480+0000 7f29ebfe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.482+0000 7f2a287a9800 20 remove_watcher() i=0 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.482+0000 7f2a287a9800 2 removed watcher, disabling cache 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.483+0000 7f2a287a9800 20 remove_watcher() i=1 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.484+0000 7f2a287a9800 20 remove_watcher() i=2 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.485+0000 7f2a287a9800 20 remove_watcher() i=3 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.486+0000 7f2a287a9800 20 remove_watcher() i=4 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.487+0000 7f2a287a9800 20 remove_watcher() i=5 2023-12-03T11:36:39.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.488+0000 7f2a287a9800 20 remove_watcher() i=6 2023-12-03T11:36:39.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.489+0000 7f2a287a9800 20 remove_watcher() i=7 2023-12-03T11:36:39.530 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.530 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:36:39.530 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:36:39.530 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:36:39.547 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:36:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:36:39.578 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:36:39.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.591+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 20 realm 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.593+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.594+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.594+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.595+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.595+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.595+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:36:39.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.595+0000 7f455f732800 20 searching for the correct realm 2023-12-03T11:36:39.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.605+0000 7f455bb63640 0 -- 172.21.15.12:0/1197953868 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5601272ffaa0 msgr2=0x560127301f60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:36:39.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.606+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 zone default found 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 4 Realm: () 2023-12-03T11:36:39.609 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 zonegroup default 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 started sync module instance, tier type = 2023-12-03T11:36:39.610 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.607+0000 7f455f732800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:36:39.615 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.614+0000 7f455bb63640 0 -- 172.21.15.12:0/1197953868 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f4534079d80 msgr2=0x7f453407c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:36:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=0 2023-12-03T11:36:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=1 2023-12-03T11:36:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=2 2023-12-03T11:36:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=3 2023-12-03T11:36:39.618 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=4 2023-12-03T11:36:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=5 2023-12-03T11:36:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.617+0000 7f455f732800 20 add_watcher() i=6 2023-12-03T11:36:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.618+0000 7f455f732800 20 add_watcher() i=7 2023-12-03T11:36:39.619 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.618+0000 7f455f732800 2 all 8 watchers are set, enabling cache 2023-12-03T11:36:39.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.619+0000 7f45217e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.620+0000 7f455f732800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:36:39.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.620+0000 7f455f732800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:36:39.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f455f732800 5 note: GC not initialized 2023-12-03T11:36:39.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f448ffff640 20 reqs_thread_entry: start 2023-12-03T11:36:39.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f455f732800 20 init_complete bucket index max shards: 1 2023-12-03T11:36:39.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f455f732800 20 Filter name: none 2023-12-03T11:36:39.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f448dffb640 20 reqs_thread_entry: start 2023-12-03T11:36:39.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f455f732800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:36:39.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.626+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : miss 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 20 rados->read ofs=0 len=0 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:36:39.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:36:39.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:36:39.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:36:39.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.627+0000 7f455f732800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:36:39.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.629+0000 7f455f732800 0 INFO: RGWReshardLock::lock found lock on abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to be held by another RGW process; skipping for now 2023-12-03T11:36:39.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.629+0000 7f45217e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:36:39.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.631+0000 7f455f732800 20 remove_watcher() i=0 2023-12-03T11:36:39.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.632+0000 7f455f732800 2 removed watcher, disabling cache 2023-12-03T11:36:39.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.633+0000 7f455f732800 20 remove_watcher() i=1 2023-12-03T11:36:39.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.633+0000 7f455f732800 20 remove_watcher() i=2 2023-12-03T11:36:39.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.635+0000 7f455f732800 20 remove_watcher() i=3 2023-12-03T11:36:39.636 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.636+0000 7f455f732800 20 remove_watcher() i=4 2023-12-03T11:36:39.637 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.636+0000 7f455f732800 20 remove_watcher() i=5 2023-12-03T11:36:39.638 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.637+0000 7f455f732800 20 remove_watcher() i=6 2023-12-03T11:36:39.640 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:36:39.639+0000 7f455f732800 20 remove_watcher() i=7 2023-12-03T11:36:39.644 INFO:tasks.workunit.client.0.smithi012.stderr:waiting 30 seconds for reshard lock to expire... 2023-12-03T11:37:09.677 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-block-writes --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:37:09.713 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:09.714 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:09.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.728+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 realm 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.730+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.731+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.732+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:09.733 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.732+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.733+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:09.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.733+0000 7f4769b32800 20 searching for the correct realm 2023-12-03T11:37:09.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.741+0000 7f4764f61640 0 -- 172.21.15.12:0/543181178 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x55fd38d54af0 msgr2=0x55fd38d56ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:09.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:09.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:09.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:09.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:09.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.745+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 zone default found 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 4 Realm: () 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:09.748 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 zonegroup default 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.746+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4764f61640 0 -- 172.21.15.12:0/543181178 >> [v2:172.21.15.191:6805/3310030881,v1:172.21.15.191:6807/3310030881] conn(0x7f475809d040 msgr2=0x7f47580b1120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4769b32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4769b32800 20 started sync module instance, tier type = 2023-12-03T11:37:09.749 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.747+0000 7f4769b32800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:09.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=0 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=1 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=2 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=3 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=4 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=5 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.758+0000 7f4769b32800 20 add_watcher() i=6 2023-12-03T11:37:09.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.759+0000 7f4769b32800 20 add_watcher() i=7 2023-12-03T11:37:09.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.759+0000 7f4769b32800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:09.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.761+0000 7f472afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:09.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.762+0000 7f4769b32800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:09.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.762+0000 7f4769b32800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:09.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.768+0000 7f4769b32800 5 note: GC not initialized 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.768+0000 7f4699ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 20 Filter name: none 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f468b7fe640 20 reqs_thread_entry: start 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 10 cache get: name=default.rgw.meta+root+abort-at-block-writes : miss 2023-12-03T11:37:09.770 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:37:09.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 10 cache put: name=default.rgw.meta+root+abort-at-block-writes info.flags=0x11 2023-12-03T11:37:09.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 10 adding default.rgw.meta+root+abort-at-block-writes to cache LRU end 2023-12-03T11:37:09.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : miss 2023-12-03T11:37:09.771 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.769+0000 7f4769b32800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 20 rados_obj.operate() r=0 bl.length=405 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:09.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.770+0000 7f4769b32800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:37:09.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.772+0000 7f4769b32800 10 init_target_layout removing existing target index objects from a previous reshard attempt 2023-12-03T11:37:09.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f4769b32800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:37:09.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f4769b32800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:37:09.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f4769b32800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:09.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f4769b32800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22, ofs0, ns] 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f47477fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725970 cookie 94546068626288 notifier 6804 bl.length()=877 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f47477fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f47477fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.776+0000 7f47477fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.777+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_sources= new_sources= 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.777+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential sources added=[] removed=[] 2023-12-03T11:37:09.778 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.777+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_dests= new_dests= 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.777+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential dests added=[] removed=[] 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.778+0000 7f4769b32800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22.2.0 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.778+0000 7f4769b32800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22.2.0 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.778+0000 7f4769b32800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22.2.1 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.778+0000 7f4769b32800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22.2.1 2023-12-03T11:37:09.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.778+0000 7f4769b32800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22.0 2023-12-03T11:37:09.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.780+0000 7f4769b32800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:09.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.780+0000 7f4769b32800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 2023-12-03T11:37:09.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4769b32800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:37:09.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4769b32800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:37:09.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4769b32800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:09.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4769b32800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22, ofs0, ns] 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4746ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725971 cookie 94546068626288 notifier 6804 bl.length()=845 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4746ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 info.flags=0x17 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4746ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-block-writes:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 to cache LRU end 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.782+0000 7f4746ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.783+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_sources= new_sources= 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.783+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential sources added=[] removed=[] 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.783+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): orig_dests= new_dests= 2023-12-03T11:37:09.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.783+0000 7f4769b32800 20 handle_bi_update(): bucket=:abort-at-block-writes[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22]): potential dests added=[] removed=[] 2023-12-03T11:37:09.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.784+0000 7f4769b32800 1 execute INFO: reshard of bucket "abort-at-block-writes" completed successfully 2023-12-03T11:37:09.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.786+0000 7f472afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:09.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.788+0000 7f4769b32800 20 remove_watcher() i=0 2023-12-03T11:37:09.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.788+0000 7f4769b32800 2 removed watcher, disabling cache 2023-12-03T11:37:09.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.789+0000 7f4769b32800 20 remove_watcher() i=1 2023-12-03T11:37:09.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.790+0000 7f4769b32800 20 remove_watcher() i=2 2023-12-03T11:37:09.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.792+0000 7f4769b32800 20 remove_watcher() i=3 2023-12-03T11:37:09.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.793+0000 7f4769b32800 20 remove_watcher() i=4 2023-12-03T11:37:09.795 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.794+0000 7f4769b32800 20 remove_watcher() i=5 2023-12-03T11:37:09.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.795+0000 7f4769b32800 20 remove_watcher() i=6 2023-12-03T11:37:09.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.796+0000 7f4769b32800 20 remove_watcher() i=7 2023-12-03T11:37:09.803 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-block-writes 2023-12-03T11:37:09.832 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:09.832 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:09.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.846+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.847+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.847+0000 7f443571e800 20 realm 2023-12-03T11:37:09.849 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.847+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.848+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.848+0000 7f443571e800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:09.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.848+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.848+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:09.850 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.848+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.849+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:09.851 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.849+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:09.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.850+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:09.852 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:09.850+0000 7f443571e800 20 searching for the correct realm 2023-12-03T11:37:10.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.062+0000 7f4433225640 0 -- 172.21.15.12:0/3510171615 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556153ac2f30 msgr2=0x556153ac5320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.064+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 zone default found 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 4 Realm: () 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 zonegroup default 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.065+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.066+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.066+0000 7f443571e800 20 started sync module instance, tier type = 2023-12-03T11:37:10.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.066+0000 7f443571e800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:10.073 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.072+0000 7f4433a26640 0 -- 172.21.15.12:0/3510171615 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f441c03eaf0 msgr2=0x7f441c01fa90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=0 2023-12-03T11:37:10.076 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=1 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=2 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=3 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=4 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=5 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.075+0000 7f443571e800 20 add_watcher() i=6 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.076+0000 7f443571e800 20 add_watcher() i=7 2023-12-03T11:37:10.077 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.076+0000 7f443571e800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:10.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.078+0000 7f43f97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.079+0000 7f443571e800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:10.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.079+0000 7f443571e800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.086+0000 7f443571e800 5 note: GC not initialized 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.086+0000 7f435ffff640 20 reqs_thread_entry: start 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f443571e800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f443571e800 20 Filter name: none 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f435dffb640 20 reqs_thread_entry: start 2023-12-03T11:37:10.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:37:10.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.087+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.088+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:37:10.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.088+0000 7f443571e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.088+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.089+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=189 2023-12-03T11:37:10.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.089+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.089+0000 7f443571e800 20 rados_obj.operate() r=0 bl.length=373 2023-12-03T11:37:10.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.089+0000 7f443571e800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.089+0000 7f443571e800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.090+0000 7f443571e800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.091+0000 7f43f97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.093+0000 7f443571e800 20 remove_watcher() i=0 2023-12-03T11:37:10.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.093+0000 7f443571e800 2 removed watcher, disabling cache 2023-12-03T11:37:10.095 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.094+0000 7f443571e800 20 remove_watcher() i=1 2023-12-03T11:37:10.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.095+0000 7f443571e800 20 remove_watcher() i=2 2023-12-03T11:37:10.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.096+0000 7f443571e800 20 remove_watcher() i=3 2023-12-03T11:37:10.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.097+0000 7f443571e800 20 remove_watcher() i=4 2023-12-03T11:37:10.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.098+0000 7f443571e800 20 remove_watcher() i=5 2023-12-03T11:37:10.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.099+0000 7f443571e800 20 remove_watcher() i=6 2023-12-03T11:37:10.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.100+0000 7f443571e800 20 remove_watcher() i=7 2023-12-03T11:37:10.106 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.106 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:10.106 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-block-writes id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:37:10.106 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.192 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with EIO injected at commit_target_layout 2023-12-03T11:37:10.192 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.409 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:37:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:10.438 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:10.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.452+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 realm 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.454+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.455+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.455+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.456+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:10.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.456+0000 7f52bff26800 20 searching for the correct realm 2023-12-03T11:37:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.465+0000 7f52bc486640 0 -- 172.21.15.12:0/1241467640 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x555bfca63e50 msgr2=0x555bfca662f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:10.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:10.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:10.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.466+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 zone default found 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 4 Realm: () 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:10.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 zonegroup default 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.467+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.468+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.468+0000 7f52bff26800 20 started sync module instance, tier type = 2023-12-03T11:37:10.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.468+0000 7f52bff26800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:10.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.474+0000 7f52bc486640 0 -- 172.21.15.12:0/1241467640 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f5294079db0 msgr2=0x7f529407c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=0 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=1 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=2 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=3 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=4 2023-12-03T11:37:10.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=5 2023-12-03T11:37:10.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.477+0000 7f52bff26800 20 add_watcher() i=6 2023-12-03T11:37:10.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.478+0000 7f52bff26800 20 add_watcher() i=7 2023-12-03T11:37:10.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.478+0000 7f52bff26800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:10.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.479+0000 7f5281fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.480+0000 7f52bff26800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:10.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.480+0000 7f52bff26800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.485+0000 7f52bff26800 5 note: GC not initialized 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.486+0000 7f51f8ff9640 20 reqs_thread_entry: start 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.486+0000 7f52bff26800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.486+0000 7f52bff26800 20 Filter name: none 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.486+0000 7f51ee7fc640 20 reqs_thread_entry: start 2023-12-03T11:37:10.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.486+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.487+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:10.488 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.487+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.487+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.487+0000 7f52bff26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.487+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.488+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:10.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.488+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.488+0000 7f52bff26800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.488+0000 7f52bff26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.488+0000 7f52bff26800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.489+0000 7f52bff26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.489+0000 7f5281fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.492+0000 7f52bff26800 20 remove_watcher() i=0 2023-12-03T11:37:10.493 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.492+0000 7f52bff26800 2 removed watcher, disabling cache 2023-12-03T11:37:10.494 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.493+0000 7f52bff26800 20 remove_watcher() i=1 2023-12-03T11:37:10.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.494+0000 7f52bff26800 20 remove_watcher() i=2 2023-12-03T11:37:10.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.495+0000 7f52bff26800 20 remove_watcher() i=3 2023-12-03T11:37:10.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.496+0000 7f52bff26800 20 remove_watcher() i=4 2023-12-03T11:37:10.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.497+0000 7f52bff26800 20 remove_watcher() i=5 2023-12-03T11:37:10.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.498+0000 7f52bff26800 20 remove_watcher() i=6 2023-12-03T11:37:10.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.499+0000 7f52bff26800 20 remove_watcher() i=7 2023-12-03T11:37:10.505 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.505 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:10.505 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:10.505 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.507 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout 2023-12-03T11:37:10.539 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:10.539 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:10.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.554+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 20 realm 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.756+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.757+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.757+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.758+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.758+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.759+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:10.760 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.759+0000 7fcba9d23800 20 searching for the correct realm 2023-12-03T11:37:10.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.768+0000 7fcba5ae5640 0 -- 172.21.15.12:0/1525640455 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55f94870e7a0 msgr2=0x55f948710b90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.770+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.772 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.773 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 zone default found 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 4 Realm: () 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 zonegroup default 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.771+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.772+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.772+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.772+0000 7fcba9d23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.772+0000 7fcba9d23800 20 started sync module instance, tier type = 2023-12-03T11:37:10.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.772+0000 7fcba9d23800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:10.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.774+0000 7fcba5ae5640 0 -- 172.21.15.12:0/1525640455 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x7fcb9c070410 msgr2=0x7fcb8007ee20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=0 2023-12-03T11:37:10.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=1 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=2 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=3 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=4 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=5 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.781+0000 7fcba9d23800 20 add_watcher() i=6 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.782+0000 7fcba9d23800 20 add_watcher() i=7 2023-12-03T11:37:10.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.782+0000 7fcba9d23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:10.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.784+0000 7fcb6b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.784+0000 7fcba9d23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:10.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.784+0000 7fcba9d23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.790+0000 7fcba9d23800 5 note: GC not initialized 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.790+0000 7fcada7fc640 20 reqs_thread_entry: start 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 20 Filter name: none 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcacbfff640 20 reqs_thread_entry: start 2023-12-03T11:37:10.792 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:37:10.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:10.793 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 : miss 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.791+0000 7fcba9d23800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:10.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.792+0000 7fcba9d23800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 2023-12-03T11:37:10.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcba9d23800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:10.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcba9d23800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:10.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcba9d23800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:10.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcba9d23800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23, ofs0, ns] 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcb87fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725928 cookie 94529150639504 notifier 6864 bl.length()=893 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcb87fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcb87fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.796+0000 7fcb87fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.797+0000 7fcba9d23800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_sources= new_sources= 2023-12-03T11:37:10.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.797+0000 7fcba9d23800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential sources added=[] removed=[] 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.797+0000 7fcba9d23800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_dests= new_dests= 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.797+0000 7fcba9d23800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential dests added=[] removed=[] 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.798+0000 7fcba9d23800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.1.0 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.798+0000 7fcba9d23800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.1.0 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.798+0000 7fcba9d23800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.1.1 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.798+0000 7fcba9d23800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.1.1 2023-12-03T11:37:10.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.798+0000 7fcba9d23800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.0 2023-12-03T11:37:10.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.800+0000 7fcba9d23800 -1 Injecting error=-5 at location=commit_target_layout 2023-12-03T11:37:10.801 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.800+0000 7fcba9d23800 0 ERROR: commit_reshard failed to commit target index layout: (5) Input/output error 2023-12-03T11:37:10.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.803+0000 7fcb6b7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.805+0000 7fcba9d23800 20 remove_watcher() i=0 2023-12-03T11:37:10.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.805+0000 7fcba9d23800 2 removed watcher, disabling cache 2023-12-03T11:37:10.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.806+0000 7fcba9d23800 20 remove_watcher() i=1 2023-12-03T11:37:10.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.807+0000 7fcba9d23800 20 remove_watcher() i=2 2023-12-03T11:37:10.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.808+0000 7fcba9d23800 20 remove_watcher() i=3 2023-12-03T11:37:10.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.809+0000 7fcba9d23800 20 remove_watcher() i=4 2023-12-03T11:37:10.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.810+0000 7fcba9d23800 20 remove_watcher() i=5 2023-12-03T11:37:10.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.811+0000 7fcba9d23800 20 remove_watcher() i=6 2023-12-03T11:37:10.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.812+0000 7fcba9d23800 20 remove_watcher() i=7 2023-12-03T11:37:10.820 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:37:10.847 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:10.847 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:10.861 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.860+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.863 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 realm 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.862+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.863+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.863+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.864+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:10.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.864+0000 7f7206137800 20 searching for the correct realm 2023-12-03T11:37:10.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.873+0000 7f7201e6f640 0 -- 172.21.15.12:0/1452766609 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5585071b5bd0 msgr2=0x5585071b7fc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.874+0000 7f7206137800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.874+0000 7f7206137800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.874+0000 7f7206137800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.874+0000 7f7206137800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.874+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:10.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:10.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 zone default found 2023-12-03T11:37:10.877 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 4 Realm: () 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 zonegroup default 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.875+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.876+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.876+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.876+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.876+0000 7f7206137800 20 started sync module instance, tier type = 2023-12-03T11:37:10.878 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.876+0000 7f7206137800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:10.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.883+0000 7f7202670640 0 -- 172.21.15.12:0/1452766609 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5585071bdcb0 msgr2=0x5585071c0170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:10.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=0 2023-12-03T11:37:10.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=1 2023-12-03T11:37:10.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=2 2023-12-03T11:37:10.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=3 2023-12-03T11:37:10.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=4 2023-12-03T11:37:10.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=5 2023-12-03T11:37:10.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.885+0000 7f7206137800 20 add_watcher() i=6 2023-12-03T11:37:10.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.886+0000 7f7206137800 20 add_watcher() i=7 2023-12-03T11:37:10.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.886+0000 7f7206137800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:10.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.888+0000 7f71c77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.888+0000 7f7206137800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:10.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.888+0000 7f7206137800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.894+0000 7f7206137800 5 note: GC not initialized 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.894+0000 7f71367fc640 20 reqs_thread_entry: start 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.894+0000 7f7206137800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.894+0000 7f7206137800 20 Filter name: none 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.895+0000 7f7127fff640 20 reqs_thread_entry: start 2023-12-03T11:37:10.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.895+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.895+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:10.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.895+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:37:10.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:10.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:37:10.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.896+0000 7f7206137800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:10.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.897+0000 7f7206137800 20 rados->read ofs=0 len=0 2023-12-03T11:37:10.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.897+0000 7f7206137800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:10.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.898+0000 7f71c77e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:10.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.900+0000 7f7206137800 20 remove_watcher() i=0 2023-12-03T11:37:10.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.900+0000 7f7206137800 2 removed watcher, disabling cache 2023-12-03T11:37:10.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.901+0000 7f7206137800 20 remove_watcher() i=1 2023-12-03T11:37:10.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.902+0000 7f7206137800 20 remove_watcher() i=2 2023-12-03T11:37:10.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.903+0000 7f7206137800 20 remove_watcher() i=3 2023-12-03T11:37:10.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.904+0000 7f7206137800 20 remove_watcher() i=4 2023-12-03T11:37:10.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.905+0000 7f7206137800 20 remove_watcher() i=5 2023-12-03T11:37:10.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.906+0000 7f7206137800 20 remove_watcher() i=6 2023-12-03T11:37:10.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:10.907+0000 7f7206137800 20 remove_watcher() i=7 2023-12-03T11:37:10.913 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.913 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:10.913 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:10.913 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:10.965 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:37:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:10.996 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:11.010 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.009+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 realm 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.011+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.013 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.012+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.012+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.013+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:11.014 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.013+0000 7f2ec6132800 20 searching for the correct realm 2023-12-03T11:37:11.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.023+0000 7f2ec1d62640 0 -- 172.21.15.12:0/2377994479 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x564bce93cc80 msgr2=0x564bce93f070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.026 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.024+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 zone default found 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 4 Realm: () 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 zonegroup default 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 started sync module instance, tier type = 2023-12-03T11:37:11.028 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.025+0000 7f2ec6132800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:11.033 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.032+0000 7f2ec2563640 0 -- 172.21.15.12:0/2377994479 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x564bce944a80 msgr2=0x564bce946e90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=0 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=1 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=2 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=3 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=4 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=5 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.034+0000 7f2ec6132800 20 add_watcher() i=6 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.035+0000 7f2ec6132800 20 add_watcher() i=7 2023-12-03T11:37:11.036 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.035+0000 7f2ec6132800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:11.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.037+0000 7f2e877e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.037+0000 7f2ec6132800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:11.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.037+0000 7f2ec6132800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:11.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.043+0000 7f2ec6132800 5 note: GC not initialized 2023-12-03T11:37:11.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2df67fc640 20 reqs_thread_entry: start 2023-12-03T11:37:11.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2ec6132800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:11.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2ec6132800 20 Filter name: none 2023-12-03T11:37:11.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2de7fff640 20 reqs_thread_entry: start 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2ec6132800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.044+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 : miss 2023-12-03T11:37:11.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.045+0000 7f2ec6132800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 2023-12-03T11:37:11.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.047+0000 7f2ec6132800 10 init_target_layout removing existing target index objects from a previous reshard attempt 2023-12-03T11:37:11.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ec6132800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:11.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ec6132800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ec6132800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ec6132800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23, ofs0, ns] 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ea3fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725929 cookie 94883588381424 notifier 6566 bl.length()=893 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ea3fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ea3fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.051+0000 7f2ea3fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.052+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_sources= new_sources= 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.052+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential sources added=[] removed=[] 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.052+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_dests= new_dests= 2023-12-03T11:37:11.053 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.052+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential dests added=[] removed=[] 2023-12-03T11:37:11.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.053+0000 7f2ec6132800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.2.0 2023-12-03T11:37:11.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.053+0000 7f2ec6132800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.2.0 2023-12-03T11:37:11.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.053+0000 7f2ec6132800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.2.1 2023-12-03T11:37:11.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.053+0000 7f2ec6132800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.2.1 2023-12-03T11:37:11.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.053+0000 7f2ec6132800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23.0 2023-12-03T11:37:11.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.055+0000 7f2ec6132800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:11.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.055+0000 7f2ec6132800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 2023-12-03T11:37:11.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ec6132800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:11.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ec6132800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:11.057 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ec6132800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ec6132800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23, ofs0, ns] 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ea37fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725930 cookie 94883588381424 notifier 6566 bl.length()=861 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ea37fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 info.flags=0x17 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ea37fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 to cache LRU end 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.056+0000 7f2ea37fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.057+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_sources= new_sources= 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.057+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential sources added=[] removed=[] 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.057+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): orig_dests= new_dests= 2023-12-03T11:37:11.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.057+0000 7f2ec6132800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23]): potential dests added=[] removed=[] 2023-12-03T11:37:11.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.057+0000 7f2ec6132800 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2023-12-03T11:37:11.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.059+0000 7f2e877e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.062+0000 7f2ec6132800 20 remove_watcher() i=0 2023-12-03T11:37:11.063 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.062+0000 7f2ec6132800 2 removed watcher, disabling cache 2023-12-03T11:37:11.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.063+0000 7f2ec6132800 20 remove_watcher() i=1 2023-12-03T11:37:11.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.064+0000 7f2ec6132800 20 remove_watcher() i=2 2023-12-03T11:37:11.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.065+0000 7f2ec6132800 20 remove_watcher() i=3 2023-12-03T11:37:11.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.066+0000 7f2ec6132800 20 remove_watcher() i=4 2023-12-03T11:37:11.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.067+0000 7f2ec6132800 20 remove_watcher() i=5 2023-12-03T11:37:11.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.068+0000 7f2ec6132800 20 remove_watcher() i=6 2023-12-03T11:37:11.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.069+0000 7f2ec6132800 20 remove_watcher() i=7 2023-12-03T11:37:11.077 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:37:11.105 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:11.105 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:11.119 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.118+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.119+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.119+0000 7f92c1b2c800 20 realm 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.119+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.120+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.120+0000 7f92c1b2c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.120+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.120+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.120+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.121+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.121+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.122+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:11.123 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.122+0000 7f92c1b2c800 20 searching for the correct realm 2023-12-03T11:37:11.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.332+0000 7f92bd8c3640 0 -- 172.21.15.12:0/1372809737 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f92b4074430 msgr2=0x7f92b405da80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:11.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:11.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:11.335 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.336 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.334+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 zone default found 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 4 Realm: () 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 zonegroup default 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.337 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.335+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.336+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.336+0000 7f92c1b2c800 20 started sync module instance, tier type = 2023-12-03T11:37:11.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.336+0000 7f92c1b2c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:11.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.341+0000 7f92bd8c3640 0 -- 172.21.15.12:0/1372809737 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f92b4087c60 msgr2=0x7f92b4086960 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.346 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=0 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=1 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=2 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=3 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=4 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=5 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.345+0000 7f92c1b2c800 20 add_watcher() i=6 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.346+0000 7f92c1b2c800 20 add_watcher() i=7 2023-12-03T11:37:11.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.346+0000 7f92c1b2c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:11.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.348+0000 7f92837e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.348+0000 7f92c1b2c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:11.349 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.348+0000 7f92c1b2c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:11.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.354+0000 7f92c1b2c800 5 note: GC not initialized 2023-12-03T11:37:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.355+0000 7f91f27fc640 20 reqs_thread_entry: start 2023-12-03T11:37:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.355+0000 7f92c1b2c800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.355+0000 7f92c1b2c800 20 Filter name: none 2023-12-03T11:37:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.355+0000 7f91e3fff640 20 reqs_thread_entry: start 2023-12-03T11:37:11.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.355+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.356+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.356+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.356+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.356+0000 7f92c1b2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.356+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.357+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.358 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.357+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.357+0000 7f92c1b2c800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.357+0000 7f92c1b2c800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.357+0000 7f92c1b2c800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.358+0000 7f92c1b2c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.359 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.358+0000 7f92837e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.361+0000 7f92c1b2c800 20 remove_watcher() i=0 2023-12-03T11:37:11.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.361+0000 7f92c1b2c800 2 removed watcher, disabling cache 2023-12-03T11:37:11.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.362+0000 7f92c1b2c800 20 remove_watcher() i=1 2023-12-03T11:37:11.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.363+0000 7f92c1b2c800 20 remove_watcher() i=2 2023-12-03T11:37:11.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.365+0000 7f92c1b2c800 20 remove_watcher() i=3 2023-12-03T11:37:11.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.366+0000 7f92c1b2c800 20 remove_watcher() i=4 2023-12-03T11:37:11.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.367+0000 7f92c1b2c800 20 remove_watcher() i=5 2023-12-03T11:37:11.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.368+0000 7f92c1b2c800 20 remove_watcher() i=6 2023-12-03T11:37:11.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.369+0000 7f92c1b2c800 20 remove_watcher() i=7 2023-12-03T11:37:11.375 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.375 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:11.375 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:37:11.375 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.437 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with ECANCELED injected at commit_target_layout 2023-12-03T11:37:11.437 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.641 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:37:11.669 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:11.669 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:11.683 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.682+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.683+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.683+0000 7f0635111800 20 realm 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.683+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.684+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.684+0000 7f0635111800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.684+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.684+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.684+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.685+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.686 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.685+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.686+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:11.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.686+0000 7f0635111800 20 searching for the correct realm 2023-12-03T11:37:11.696 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.695+0000 7f0632c18640 0 -- 172.21.15.12:0/594135828 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55e2143c36e0 msgr2=0x55e2143c5ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:11.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:11.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:11.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:11.698 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.699 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.697+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 zone default found 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 4 Realm: () 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 zonegroup default 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.700 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 started sync module instance, tier type = 2023-12-03T11:37:11.701 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.698+0000 7f0635111800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:11.706 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.705+0000 7f0633419640 0 -- 172.21.15.12:0/594135828 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55e2143cb7c0 msgr2=0x55e2143cdc80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=0 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=1 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=2 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=3 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=4 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.707+0000 7f0635111800 20 add_watcher() i=5 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.708+0000 7f0635111800 20 add_watcher() i=6 2023-12-03T11:37:11.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.708+0000 7f0635111800 20 add_watcher() i=7 2023-12-03T11:37:11.710 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.708+0000 7f0635111800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:11.711 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.710+0000 7f0573fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.710+0000 7f0635111800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:11.712 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.710+0000 7f0635111800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:11.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f0635111800 5 note: GC not initialized 2023-12-03T11:37:11.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f056affd640 20 reqs_thread_entry: start 2023-12-03T11:37:11.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f0635111800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:11.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f0635111800 20 Filter name: none 2023-12-03T11:37:11.718 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f0568ff9640 20 reqs_thread_entry: start 2023-12-03T11:37:11.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.717+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.718+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.719 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.718+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.719+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.719+0000 7f0635111800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.719+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.719+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.720 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.719+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.720+0000 7f0635111800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.720+0000 7f0635111800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.720+0000 7f0635111800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.720+0000 7f0635111800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.722 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.721+0000 7f0573fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.723+0000 7f0635111800 20 remove_watcher() i=0 2023-12-03T11:37:11.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.723+0000 7f0635111800 2 removed watcher, disabling cache 2023-12-03T11:37:11.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.724+0000 7f0635111800 20 remove_watcher() i=1 2023-12-03T11:37:11.726 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.725+0000 7f0635111800 20 remove_watcher() i=2 2023-12-03T11:37:11.728 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.727+0000 7f0635111800 20 remove_watcher() i=3 2023-12-03T11:37:11.729 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.728+0000 7f0635111800 20 remove_watcher() i=4 2023-12-03T11:37:11.730 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.729+0000 7f0635111800 20 remove_watcher() i=5 2023-12-03T11:37:11.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.730+0000 7f0635111800 20 remove_watcher() i=6 2023-12-03T11:37:11.731 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.731+0000 7f0635111800 20 remove_watcher() i=7 2023-12-03T11:37:11.736 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.736 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:11.737 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:11.737 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.738 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at commit_target_layout --inject-error-code 125 2023-12-03T11:37:11.773 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:11.773 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:11.787 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.786+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 realm 2023-12-03T11:37:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.789 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:11.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.788+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.789+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.790 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.789+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.790+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:11.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.790+0000 7ffb65f3a800 20 searching for the correct realm 2023-12-03T11:37:11.800 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.799+0000 7ffb61c06640 0 -- 172.21.15.12:0/3651879592 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7ffb580700b0 msgr2=0x7ffb5805d860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.801+0000 7ffb65f3a800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.801+0000 7ffb65f3a800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.801+0000 7ffb65f3a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.801+0000 7ffb65f3a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.802+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.802+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.803 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.802+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.802+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.802+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 zone default found 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 4 Realm: () 2023-12-03T11:37:11.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 zonegroup default 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.803+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.804+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.804+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.804+0000 7ffb65f3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.804+0000 7ffb65f3a800 20 started sync module instance, tier type = 2023-12-03T11:37:11.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.804+0000 7ffb65f3a800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:11.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.809+0000 7ffb61405640 0 -- 172.21.15.12:0/3651879592 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x557dd170d440 msgr2=0x557dd170f830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=0 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=1 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=2 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=3 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=4 2023-12-03T11:37:11.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=5 2023-12-03T11:37:11.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.813+0000 7ffb65f3a800 20 add_watcher() i=6 2023-12-03T11:37:11.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.814+0000 7ffb65f3a800 20 add_watcher() i=7 2023-12-03T11:37:11.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.814+0000 7ffb65f3a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:11.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.815+0000 7ffb277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.816+0000 7ffb65f3a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:11.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.816+0000 7ffb65f3a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.822+0000 7ffb65f3a800 5 note: GC not initialized 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.822+0000 7ffa967fc640 20 reqs_thread_entry: start 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.822+0000 7ffb65f3a800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.822+0000 7ffb65f3a800 20 Filter name: none 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.822+0000 7ffa87fff640 20 reqs_thread_entry: start 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 cache get: name=default.rgw.meta+root+error-at-commit-target-layout : miss 2023-12-03T11:37:11.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 cache put: name=default.rgw.meta+root+error-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 adding default.rgw.meta+root+error-at-commit-target-layout to cache LRU end 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 : miss 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 info.flags=0x17 2023-12-03T11:37:11.825 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 adding default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 to cache LRU end 2023-12-03T11:37:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.823+0000 7ffb65f3a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 2023-12-03T11:37:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 info.flags=0x17 2023-12-03T11:37:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 to cache LRU end 2023-12-03T11:37:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24, ofs0, ns] 2023-12-03T11:37:11.829 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb43fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725933 cookie 93998873108016 notifier 6629 bl.length()=893 2023-12-03T11:37:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb43fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 info.flags=0x17 2023-12-03T11:37:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb43fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 to cache LRU end 2023-12-03T11:37:11.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb43fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): orig_sources= new_sources= 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): potential sources added=[] removed=[] 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): orig_dests= new_dests= 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.828+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): potential dests added=[] removed=[] 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.829+0000 7ffb65f3a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24.1.0 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.829+0000 7ffb65f3a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24.1.0 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.829+0000 7ffb65f3a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24.1.1 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.829+0000 7ffb65f3a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24.1.1 2023-12-03T11:37:11.831 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.829+0000 7ffb65f3a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24.0 2023-12-03T11:37:11.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.832+0000 7ffb65f3a800 -1 Injecting error=-125 at location=commit_target_layout 2023-12-03T11:37:11.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.832+0000 7ffb65f3a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:11.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.832+0000 7ffb65f3a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb65f3a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 info.flags=0x17 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb65f3a800 10 moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 to cache LRU end 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb65f3a800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb65f3a800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24, ofs0, ns] 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb43fff640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725934 cookie 93998873108016 notifier 6629 bl.length()=861 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb43fff640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 info.flags=0x17 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb43fff640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 to cache LRU end 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.833+0000 7ffb43fff640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:11.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.834+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): orig_sources= new_sources= 2023-12-03T11:37:11.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.834+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): potential sources added=[] removed=[] 2023-12-03T11:37:11.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.834+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): orig_dests= new_dests= 2023-12-03T11:37:11.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.834+0000 7ffb65f3a800 20 handle_bi_update(): bucket=:error-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24]): potential dests added=[] removed=[] 2023-12-03T11:37:11.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.835+0000 7ffb65f3a800 1 execute INFO: reshard of bucket "error-at-commit-target-layout" completed successfully 2023-12-03T11:37:11.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.837+0000 7ffb277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.839+0000 7ffb65f3a800 20 remove_watcher() i=0 2023-12-03T11:37:11.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.839+0000 7ffb65f3a800 2 removed watcher, disabling cache 2023-12-03T11:37:11.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.840+0000 7ffb65f3a800 20 remove_watcher() i=1 2023-12-03T11:37:11.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.841+0000 7ffb65f3a800 20 remove_watcher() i=2 2023-12-03T11:37:11.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.842+0000 7ffb65f3a800 20 remove_watcher() i=3 2023-12-03T11:37:11.844 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.843+0000 7ffb65f3a800 20 remove_watcher() i=4 2023-12-03T11:37:11.845 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.844+0000 7ffb65f3a800 20 remove_watcher() i=5 2023-12-03T11:37:11.846 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.845+0000 7ffb65f3a800 20 remove_watcher() i=6 2023-12-03T11:37:11.847 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.846+0000 7ffb65f3a800 20 remove_watcher() i=7 2023-12-03T11:37:11.853 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-commit-target-layout 2023-12-03T11:37:11.881 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:11.881 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:11.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.894+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 realm 2023-12-03T11:37:11.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.896+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.897+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.897+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.898+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:11.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.898+0000 7f3866124800 20 searching for the correct realm 2023-12-03T11:37:11.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.907+0000 7f386267f640 0 -- 172.21.15.12:0/3367158883 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55b94fe3d870 msgr2=0x55b94fe3fd00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.908+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:11.910 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:11.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 zone default found 2023-12-03T11:37:11.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 4 Realm: () 2023-12-03T11:37:11.911 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 zonegroup default 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.909+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.910+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.910+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.910+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.910+0000 7f3866124800 20 started sync module instance, tier type = 2023-12-03T11:37:11.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.910+0000 7f3866124800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:11.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.914+0000 7f3861e7e640 0 -- 172.21.15.12:0/3367158883 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f384c02fb80 msgr2=0x7f384c017970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:11.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=0 2023-12-03T11:37:11.920 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=1 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=2 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=3 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=4 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=5 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.919+0000 7f3866124800 20 add_watcher() i=6 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.920+0000 7f3866124800 20 add_watcher() i=7 2023-12-03T11:37:11.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.920+0000 7f3866124800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:11.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.921+0000 7f38277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.922+0000 7f3866124800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:11.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.922+0000 7f3866124800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f3866124800 5 note: GC not initialized 2023-12-03T11:37:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f37967fc640 20 reqs_thread_entry: start 2023-12-03T11:37:11.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f3866124800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f3866124800 20 Filter name: none 2023-12-03T11:37:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f3787fff640 20 reqs_thread_entry: start 2023-12-03T11:37:11.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.928+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.929+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.929+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.929+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.929+0000 7f3866124800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.929+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.930+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:11.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.930+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.930+0000 7f3866124800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.930+0000 7f3866124800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.931+0000 7f3866124800 20 rados->read ofs=0 len=0 2023-12-03T11:37:11.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.931+0000 7f3866124800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:11.933 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.932+0000 7f38277e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:11.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.934+0000 7f3866124800 20 remove_watcher() i=0 2023-12-03T11:37:11.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.934+0000 7f3866124800 2 removed watcher, disabling cache 2023-12-03T11:37:11.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.935+0000 7f3866124800 20 remove_watcher() i=1 2023-12-03T11:37:11.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.936+0000 7f3866124800 20 remove_watcher() i=2 2023-12-03T11:37:11.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.937+0000 7f3866124800 20 remove_watcher() i=3 2023-12-03T11:37:11.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.938+0000 7f3866124800 20 remove_watcher() i=4 2023-12-03T11:37:11.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.939+0000 7f3866124800 20 remove_watcher() i=5 2023-12-03T11:37:11.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.940+0000 7f3866124800 20 remove_watcher() i=6 2023-12-03T11:37:11.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:11.941+0000 7f3866124800 20 remove_watcher() i=7 2023-12-03T11:37:11.946 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:11.947 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:11.947 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24 num_objects: 20 size_kb: 1 num_shards: 2 2023-12-03T11:37:11.947 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.034 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with abort at commit_target_layout 2023-12-03T11:37:12.034 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.284 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:37:12.311 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:12.311 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:12.326 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.325+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 realm 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.328 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.327+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.328+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.329 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.328+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.329+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:12.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.329+0000 7f3f21721800 20 searching for the correct realm 2023-12-03T11:37:12.339 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.338+0000 7f3f1d4e5640 0 -- 172.21.15.12:0/4212172157 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55e4f656f7c0 msgr2=0x55e4f6571bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.339+0000 7f3f21721800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.339+0000 7f3f21721800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.339+0000 7f3f21721800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.339+0000 7f3f21721800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.339+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 zone default found 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 4 Realm: () 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 zonegroup default 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.340+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.341+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.341+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.341+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.341+0000 7f3f21721800 20 started sync module instance, tier type = 2023-12-03T11:37:12.343 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.341+0000 7f3f21721800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:12.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.346+0000 7f3f1dce6640 0 -- 172.21.15.12:0/4212172157 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f3f0803c440 msgr2=0x7f3f0803c8a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=0 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=1 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=2 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=3 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=4 2023-12-03T11:37:12.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=5 2023-12-03T11:37:12.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.350+0000 7f3f21721800 20 add_watcher() i=6 2023-12-03T11:37:12.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.351+0000 7f3f21721800 20 add_watcher() i=7 2023-12-03T11:37:12.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.351+0000 7f3f21721800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:12.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.353+0000 7f3ee2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.353+0000 7f3f21721800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:12.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.353+0000 7f3f21721800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.359+0000 7f3f21721800 5 note: GC not initialized 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.359+0000 7f3e51ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.359+0000 7f3f21721800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.359+0000 7f3f21721800 20 Filter name: none 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.360+0000 7f3e437fe640 20 reqs_thread_entry: start 2023-12-03T11:37:12.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.360+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.360+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.360+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.361+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.361+0000 7f3f21721800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.361+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.361+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.361+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.362+0000 7f3f21721800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:12.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.362+0000 7f3f21721800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:12.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.362+0000 7f3f21721800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.362+0000 7f3f21721800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.363+0000 7f3ee2fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.365+0000 7f3f21721800 20 remove_watcher() i=0 2023-12-03T11:37:12.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.365+0000 7f3f21721800 2 removed watcher, disabling cache 2023-12-03T11:37:12.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.366+0000 7f3f21721800 20 remove_watcher() i=1 2023-12-03T11:37:12.368 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.367+0000 7f3f21721800 20 remove_watcher() i=2 2023-12-03T11:37:12.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.368+0000 7f3f21721800 20 remove_watcher() i=3 2023-12-03T11:37:12.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.369+0000 7f3f21721800 20 remove_watcher() i=4 2023-12-03T11:37:12.371 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.370+0000 7f3f21721800 20 remove_watcher() i=5 2023-12-03T11:37:12.372 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.371+0000 7f3f21721800 20 remove_watcher() i=6 2023-12-03T11:37:12.373 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.372+0000 7f3f21721800 20 remove_watcher() i=7 2023-12-03T11:37:12.378 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:12.378 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:12.378 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.380 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at commit_target_layout 2023-12-03T11:37:12.412 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:12.413 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:12.427 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.426+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 realm 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:12.429 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.429+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.430 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.429+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.430+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:12.431 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.430+0000 7f12af924800 20 searching for the correct realm 2023-12-03T11:37:12.441 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.440+0000 7f12ab67e640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556b01c14100 msgr2=0x556b01c164f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:12.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 zone default found 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 4 Realm: () 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 zonegroup default 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.443+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.443+0000 7f12af924800 20 started sync module instance, tier type = 2023-12-03T11:37:12.445 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.443+0000 7f12af924800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:12.450 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.449+0000 7f12abe7f640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556b01c1c1f0 msgr2=0x556b01c1e6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=0 2023-12-03T11:37:12.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=1 2023-12-03T11:37:12.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=2 2023-12-03T11:37:12.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=3 2023-12-03T11:37:12.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=4 2023-12-03T11:37:12.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=5 2023-12-03T11:37:12.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=6 2023-12-03T11:37:12.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.453+0000 7f12af924800 20 add_watcher() i=7 2023-12-03T11:37:12.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.453+0000 7f12af924800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:12.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.454+0000 7f12717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:12.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:12.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f12af924800 5 note: GC not initialized 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f11dffff640 20 reqs_thread_entry: start 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f12af924800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f12af924800 20 Filter name: none 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f11ddffb640 20 reqs_thread_entry: start 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.461+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.462+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.462+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.462+0000 7f12af924800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.462+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 : miss 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.462+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:12.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.463+0000 7f12af924800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 2023-12-03T11:37:12.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12af924800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12af924800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25, ofs0, ns] 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725937 cookie 93918079364640 notifier 6689 bl.length()=893 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_sources= new_sources= 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential sources added=[] removed=[] 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_dests= new_dests= 2023-12-03T11:37:12.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential dests added=[] removed=[] 2023-12-03T11:37:12.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.0 2023-12-03T11:37:12.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f12af924800 time 2023-12-03T11:37:12.472362+0000 2023-12-03T11:37:12.472 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f12b256eef8] 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f12af924800 thread_name:radosgw-admin 2023-12-03T11:37:12.473 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.472+0000 7f12af924800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f12af924800 time 2023-12-03T11:37:12.472362+0000 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f12b256eef8] 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:12.474 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f12afe54db0] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f12afea154c] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f12b256ef52] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.491 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.472+0000 7f12af924800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f12af924800 thread_name:radosgw-admin 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f12afe54db0] 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f12afea154c] 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f12b256ef52] 2023-12-03T11:37:12.492 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:37:12.420+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: get_monmap_and_config 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:37:12.420+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.493 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.494 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:37:12.421+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.495 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.496 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:37:12.422+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0acbd178 EEXIST 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01bf2cc0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01bf4f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01bc56c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.604675s 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01bf4f90 auth_method 0 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.497 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01bf2cc0 auth_method 0 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01bc56c0 auth_method 0 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.498 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7029 payload 274 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:37:12.422+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f1298069820 auth_method 0 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.499 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: dump: 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.500 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:37:12.422+0000 7f12af924800 5 monclient: authenticate success, global_id 7029 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:37:12.422+0000 7f12af924800 4 set_mon_vals no callback set 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: dump: 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.501 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: shutdown 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:37:12.423+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.502 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.503 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:37:12.423+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:37:12.423+0000 7f12af924800 1 librados: starting msgr at 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:37:12.423+0000 7f12af924800 1 librados: starting objecter 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:37:12.424+0000 7f12af924800 5 asok(0x556b019baa10) register_command objecter_requests cmddesc objecter_requests hook 0x556b01bc3b50 EEXIST 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:37:12.424+0000 7f12af924800 1 librados: setting wanted keys 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:37:12.424+0000 7f12af924800 1 librados: calling monclient init 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.504 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:37:12.424+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:37:12.424+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:37:12.424+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x556b01bc2270 EEXIST 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01bc56c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01bf2cc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.505 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01bf4f90 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.607317s 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01bc56c0 auth_method 0 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01bf2cc0 auth_method 0 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.506 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01bf4f90 auth_method 0 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 6689 payload 824 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:37:12.425+0000 7f12ab67e640 1 monclient: found mon.noname-c 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: dump: 2023-12-03T11:37:12.507 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:37:12.425+0000 7f1291ffb640 4 set_mon_vals no callback set 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: dump: 2023-12-03T11:37:12.508 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:37:12.425+0000 7f12af924800 5 monclient: authenticate success, global_id 6689 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:37:12.425+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.509 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:37:12.426+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:37:12.427+0000 7f12aae7d640 10 monclient: get_auth_request con 0x7f1284069850 auth_method 0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:37:12.427+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c103d0 auth_method 0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:37:12.428+0000 7f12af924800 20 realm 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:37:12.428+0000 7f12af924800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.510 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:37:12.428+0000 7f12ab67e640 10 monclient: get_auth_request con 0x556b01c14100 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:37:12.429+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:37:12.429+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:37:12.429+0000 7f12aae7d640 10 monclient: get_auth_request con 0x556b01c178d0 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:37:12.430+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:37:12.430+0000 7f12af924800 20 searching for the correct realm 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:37:12.430+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c1c1f0 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:37:12.431+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f12a000fbb0 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:37:12.432+0000 7f12aae7d640 10 monclient: get_auth_request con 0x7f129c07c060 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:37:12.434+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f12a00598e0 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:37:12.436+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f12a005da00 auth_method 0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:37:12.440+0000 7f12ab67e640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556b01c14100 msgr2=0x556b01c164f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:37:12.440+0000 7f128bfff640 1 client.6689.objecter ms_handle_reset 0x556b01c14100 session 0x556b01983320 osd.5 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:37:12.440+0000 7f128bfff640 10 monclient: _renew_subs 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:37:12.440+0000 7f128bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.511 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:37:12.440+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f1284079db0 auth_method 0 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:37:12.442+0000 7f12af924800 20 zone default found 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:37:12.442+0000 7f12af924800 4 Realm: () 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:37:12.442+0000 7f12af924800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:37:12.442+0000 7f12af924800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:37:12.442+0000 7f12af924800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:12.512 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:37:12.442+0000 7f12af924800 20 zonegroup default 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:37:12.443+0000 7f12af924800 20 started sync module instance, tier type = 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:37:12.443+0000 7f12af924800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:37:12.449+0000 7f12abe7f640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556b01c1c1f0 msgr2=0x556b01c1e6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:37:12.449+0000 7f128bfff640 1 client.6689.objecter ms_handle_reset 0x556b01c1c1f0 session 0x556b01c1bb70 osd.3 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:37:12.449+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c14100 auth_method 0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=0 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=1 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=2 2023-12-03T11:37:12.513 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=3 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=4 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=5 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=6 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:37:12.453+0000 7f12af924800 20 add_watcher() i=7 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:37:12.453+0000 7f12af924800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.514 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:37:12.454+0000 7f12717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache list hook 0x556b01b9d7b0 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache inspect hook 0x556b01b9d7b0 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache erase hook 0x556b01b9d7b0 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache zap hook 0x556b01b9d7b0 2023-12-03T11:37:12.515 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: get_monmap_and_config 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:37:12.455+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.516 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.517 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.518 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:37:12.456+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:37:12.456+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0acbd6f8 EEXIST 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01c58640 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01c4ef80 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01c5f710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.519 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.638851s 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01c4ef80 auth_method 0 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01c5f710 auth_method 0 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01c58640 auth_method 0 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.520 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7035 payload 274 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:37:12.456+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.521 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: dump: 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.522 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:37:12.456+0000 7f12af924800 5 monclient: authenticate success, global_id 7035 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:37:12.456+0000 7f12af924800 4 set_mon_vals no callback set 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: dump: 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.523 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: shutdown 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:37:12.457+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.524 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.525 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: starting msgr at 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: starting objecter 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:37:12.458+0000 7f12af924800 5 asok(0x556b019baa10) register_command objecter_requests cmddesc objecter_requests hook 0x556b01c2a090 EEXIST 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: setting wanted keys 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: calling monclient init 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.526 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:37:12.458+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x556b01c26940 EEXIST 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:12.527 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01c4ef80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01c58640 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01c5f710 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.641351s 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01c58640 auth_method 0 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01c4ef80 auth_method 0 2023-12-03T11:37:12.528 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01c5f710 auth_method 0 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.529 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7038 payload 824 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:37:12.459+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: dump: 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.530 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: dump: 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.531 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:37:12.459+0000 7f12af924800 5 monclient: authenticate success, global_id 7038 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:37:12.459+0000 7f1270fe1640 4 set_mon_vals no callback set 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:37:12.459+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:37:12.459+0000 7f12af924800 5 asok(0x556b019baa10) register_command cr dump hook 0x556b01c621c8 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.532 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:37:12.461+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f11e0069820 auth_method 0 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:37:12.461+0000 7f12af924800 5 note: GC not initialized 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:37:12.461+0000 7f11dffff640 20 reqs_thread_entry: start 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace show hook 0x7f129c004720 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace history hook 0x7f129c004720 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace active hook 0x7f129c004720 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace active_short hook 0x7f129c004720 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:37:12.461+0000 7f12af924800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:37:12.461+0000 7f12af924800 20 Filter name: none 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:37:12.461+0000 7f11ddffb640 20 reqs_thread_entry: start 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:37:12.461+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:37:12.461+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:37:12.462+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:37:12.462+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:37:12.462+0000 7f12af924800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:37:12.462+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 : miss 2023-12-03T11:37:12.533 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:37:12.462+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:37:12.463+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:37:12.463+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:37:12.463+0000 7f12af924800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:37:12.463+0000 7f12af924800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:37:12.463+0000 7f12aae7d640 10 monclient: get_auth_request con 0x556b01ca6290 auth_method 0 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:37:12.467+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:37:12.467+0000 7f12af924800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:37:12.467+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:37:12.467+0000 7f12af924800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25, ofs0, ns] 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725937 cookie 93918079364640 notifier 6689 bl.length()=893 2023-12-03T11:37:12.534 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_sources= new_sources= 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential sources added=[] removed=[] 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_dests= new_dests= 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential dests added=[] removed=[] 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.0 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:37:12.472+0000 7f12af924800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f12af924800 time 2023-12-03T11:37:12.472362+0000 2023-12-03T11:37:12.535 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f12b256eef8] 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:37:12.472+0000 7f12af924800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f12af924800 thread_name:radosgw-admin 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.536 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f12afe54db0] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f12afea154c] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f12b256ef52] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.537 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:37:12.538 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:37:12.539 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:37:12.540 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:37:12.541 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:37:12.542 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:37:12.543 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11ddffb640 / http_manager 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11dffff640 / http_manager 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11eaffd640 / ms_dispatch 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1270fe1640 / io_context_pool 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12717e2640 / rgw_dt_lg_renew 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f128bfff640 / ms_dispatch 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12917fa640 / io_context_pool 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1291ffb640 / io_context_pool 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12a8e79640 / ms_dispatch 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12aae7d640 / msgr-worker-2 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12ab67e640 / msgr-worker-1 2023-12-03T11:37:12.544 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12abe7f640 / msgr-worker-0 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12af924800 / radosgw-admin 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/log/ceph/ceph-client.admin.103489.log 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:37:12.420+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: get_monmap_and_config 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:37:12.420+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:37:12.420+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.545 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding auth protocol: cephx 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.546 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: crc 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:37:12.420+0000 7f12af924800 5 AuthRegistry(0x556b01bc3f60) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:37:12.421+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding auth protocol: cephx 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.547 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: crc 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:37:12.421+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd030) adding con mode: secure 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:37:12.421+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:37:12.422+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0acbd178 EEXIST 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01bf2cc0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.548 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01bf4f90 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01bc56c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.604675s 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01bf4f90 auth_method 0 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01bf2cc0 auth_method 0 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.549 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01bc56c0 auth_method 0 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.550 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:37:12.422+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:37:12.422+0000 7f12a8e79640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7029 payload 274 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:37:12.422+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:37:12.422+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:37:12.422+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f1298069820 auth_method 0 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: dump: 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.551 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:37:12.422+0000 7f12af924800 5 monclient: authenticate success, global_id 7029 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:37:12.422+0000 7f12af924800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:37:12.422+0000 7f12af924800 4 set_mon_vals no callback set 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.552 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:37:12.422+0000 7f1291ffb640 10 monclient: dump: 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: shutdown 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:37:12.423+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.553 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:37:12.423+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding auth protocol: cephx 2023-12-03T11:37:12.554 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: crc 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:37:12.423+0000 7f12af924800 5 AuthRegistry(0x556b01bc85a0) adding con mode: secure 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:37:12.423+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:37:12.423+0000 7f12af924800 1 librados: starting msgr at 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:37:12.423+0000 7f12af924800 1 librados: starting objecter 2023-12-03T11:37:12.555 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:37:12.424+0000 7f12af924800 5 asok(0x556b019baa10) register_command objecter_requests cmddesc objecter_requests hook 0x556b01bc3b50 EEXIST 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:37:12.424+0000 7f12af924800 1 librados: setting wanted keys 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:37:12.424+0000 7f12af924800 1 librados: calling monclient init 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding auth protocol: cephx 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.556 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: crc 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:37:12.424+0000 7f12af924800 5 AuthRegistry(0x556b01bc2128) adding con mode: secure 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:37:12.424+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:37:12.424+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:37:12.424+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x556b01bc2270 EEXIST 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01bc56c0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01bf2cc0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01bf4f90 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.557 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:37:12.424+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.607317s 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01bc56c0 auth_method 0 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01bf2cc0 auth_method 0 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.558 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:37:12.424+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:37:12.424+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01bf4f90 auth_method 0 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:37:12.425+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 6689 payload 824 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:37:12.425+0000 7f12ab67e640 1 monclient: found mon.noname-c 2023-12-03T11:37:12.559 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:37:12.425+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: dump: 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.560 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:37:12.425+0000 7f1291ffb640 4 set_mon_vals no callback set 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:37:12.425+0000 7f128bfff640 10 monclient: dump: 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.561 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:37:12.425+0000 7f12af924800 5 monclient: authenticate success, global_id 6689 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:37:12.425+0000 7f12af924800 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:37:12.425+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:37:12.426+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:37:12.427+0000 7f128bfff640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:37:12.427+0000 7f12aae7d640 10 monclient: get_auth_request con 0x7f1284069850 auth_method 0 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:37:12.427+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c103d0 auth_method 0 2023-12-03T11:37:12.562 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:37:12.428+0000 7f12af924800 20 realm 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:37:12.428+0000 7f12af924800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:37:12.428+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:37:12.428+0000 7f12ab67e640 10 monclient: get_auth_request con 0x556b01c14100 auth_method 0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:37:12.429+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:37:12.429+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:37:12.429+0000 7f12aae7d640 10 monclient: get_auth_request con 0x556b01c178d0 auth_method 0 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:37:12.430+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:37:12.430+0000 7f12af924800 20 searching for the correct realm 2023-12-03T11:37:12.563 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:37:12.430+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c1c1f0 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:37:12.431+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f12a000fbb0 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:37:12.432+0000 7f12aae7d640 10 monclient: get_auth_request con 0x7f129c07c060 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:37:12.434+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f12a00598e0 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:37:12.436+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f12a005da00 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:37:12.440+0000 7f12ab67e640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556b01c14100 msgr2=0x556b01c164f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:37:12.440+0000 7f128bfff640 1 client.6689.objecter ms_handle_reset 0x556b01c14100 session 0x556b01983320 osd.5 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:37:12.440+0000 7f128bfff640 10 monclient: _renew_subs 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:37:12.440+0000 7f128bfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.12:3301/0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:37:12.440+0000 7f12ab67e640 10 monclient: get_auth_request con 0x7f1284079db0 auth_method 0 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:37:12.441+0000 7f12af924800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:12.564 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:37:12.441+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:37:12.442+0000 7f12af924800 20 zone default found 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:37:12.442+0000 7f12af924800 4 Realm: () 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:37:12.442+0000 7f12af924800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:37:12.442+0000 7f12af924800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:37:12.442+0000 7f12af924800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:37:12.442+0000 7f12af924800 20 zonegroup default 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.565 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:37:12.442+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:37:12.443+0000 7f12af924800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:37:12.443+0000 7f12af924800 20 started sync module instance, tier type = 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:37:12.443+0000 7f12af924800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:37:12.449+0000 7f12abe7f640 0 -- 172.21.15.12:0/1730399359 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556b01c1c1f0 msgr2=0x556b01c1e6b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:37:12.449+0000 7f128bfff640 1 client.6689.objecter ms_handle_reset 0x556b01c1c1f0 session 0x556b01c1bb70 osd.3 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:37:12.449+0000 7f12abe7f640 10 monclient: get_auth_request con 0x556b01c14100 auth_method 0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=0 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=1 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=2 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=3 2023-12-03T11:37:12.566 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=4 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=5 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:37:12.452+0000 7f12af924800 20 add_watcher() i=6 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:37:12.453+0000 7f12af924800 20 add_watcher() i=7 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:37:12.453+0000 7f12af924800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding auth protocol: cephx 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.567 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:37:12.454+0000 7f12717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: crc 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:37:12.454+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbda60) adding con mode: secure 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:37:12.455+0000 7f12af924800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache list hook 0x556b01b9d7b0 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache inspect hook 0x556b01b9d7b0 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache erase hook 0x556b01b9d7b0 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:37:12.455+0000 7f12af924800 5 asok(0x556b019baa10) register_command cache zap hook 0x556b01b9d7b0 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: get_monmap_and_config 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:37:12.455+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.568 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding auth protocol: cephx 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.569 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: crc 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x556b01c28830) adding con mode: secure 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:37:12.455+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding auth protocol: cephx 2023-12-03T11:37:12.570 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: crc 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:37:12.455+0000 7f12af924800 5 AuthRegistry(0x7ffd0acbd5b0) adding con mode: secure 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:37:12.455+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:37:12.456+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:37:12.456+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x7ffd0acbd6f8 EEXIST 2023-12-03T11:37:12.571 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01c58640 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01c4ef80 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01c5f710 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.638851s 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01c4ef80 auth_method 0 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01c5f710 auth_method 0 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.572 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01c58640 auth_method 0 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:37:12.456+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:37:12.456+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7035 payload 274 2023-12-03T11:37:12.573 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:37:12.456+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:37:12.456+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: dump: 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.574 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:37:12.456+0000 7f12af924800 5 monclient: authenticate success, global_id 7035 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:37:12.456+0000 7f12af924800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:37:12.456+0000 7f12af924800 4 set_mon_vals no callback set 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:37:12.456+0000 7f1270fe1640 10 monclient: dump: 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.575 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: shutdown 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: build_initial_monmap 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:37:12.457+0000 7f12af924800 1 build_initial for_mkfs: 0 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:37:12.457+0000 7f12af924800 10 monclient: monmap: 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:12.576 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding auth protocol: cephx 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.577 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: crc 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:37:12.457+0000 7f12af924800 5 AuthRegistry(0x556b01c5efa0) adding con mode: secure 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: starting msgr at 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: starting objecter 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:37:12.458+0000 7f12af924800 5 asok(0x556b019baa10) register_command objecter_requests cmddesc objecter_requests hook 0x556b01c2a090 EEXIST 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: setting wanted keys 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:37:12.458+0000 7f12af924800 1 librados: calling monclient init 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: init 2023-12-03T11:37:12.578 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding auth protocol: cephx 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: crc 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:37:12.458+0000 7f12af924800 5 AuthRegistry(0x556b01c267f8) adding con mode: secure 2023-12-03T11:37:12.579 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:37:12.458+0000 7f12af924800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:37:12.458+0000 7f12af924800 5 asok(0x556b019baa10) register_command rotate-key cmddesc rotate-key hook 0x556b01c26940 EEXIST 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-c con 0x556b01c4ef80 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-a con 0x556b01c58640 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): picked mon.noname-b con 0x556b01c5f710 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:37:12.458+0000 7f12af924800 10 monclient(hunting): authenticate will time out at 1642.641351s 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): get_auth_request con 0x556b01c58640 auth_method 0 2023-12-03T11:37:12.580 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): get_auth_request con 0x556b01c4ef80 auth_method 0 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:37:12.458+0000 7f12ab67e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): get_auth_request con 0x556b01c5f710 auth_method 0 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:37:12.458+0000 7f12aae7d640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:12.581 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:37:12.458+0000 7f12abe7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:37:12.459+0000 7f12aae7d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient(hunting): handle_auth_done global_id 7038 payload 824 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient: _finish_hunting 0 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:37:12.459+0000 7f12ab67e640 1 monclient: found mon.noname-a 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:37:12.459+0000 7f12ab67e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: dump: 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.582 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: _finish_auth 0 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: _check_auth_tickets 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:37:12.459+0000 7f11eaffd640 10 monclient: dump: 2023-12-03T11:37:12.583 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:37:12.459+0000 7f12af924800 5 monclient: authenticate success, global_id 7038 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:37:12.459+0000 7f1270fe1640 4 set_mon_vals no callback set 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _renew_subs 2023-12-03T11:37:12.584 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:37:12.459+0000 7f12af924800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:37:12.459+0000 7f12af924800 1 librados: init done 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:37:12.459+0000 7f12af924800 5 asok(0x556b019baa10) register_command cr dump hook 0x556b01c621c8 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:37:12.461+0000 7f11eaffd640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:37:12.461+0000 7f12abe7f640 10 monclient: get_auth_request con 0x7f11e0069820 auth_method 0 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:37:12.461+0000 7f12af924800 5 note: GC not initialized 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:37:12.461+0000 7f11dffff640 20 reqs_thread_entry: start 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace show hook 0x7f129c004720 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace history hook 0x7f129c004720 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace active hook 0x7f129c004720 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:37:12.461+0000 7f12af924800 5 asok(0x556b019baa10) register_command sync trace active_short hook 0x7f129c004720 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:37:12.461+0000 7f12af924800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:12.585 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:37:12.461+0000 7f12af924800 20 Filter name: none 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:37:12.461+0000 7f11ddffb640 20 reqs_thread_entry: start 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:37:12.461+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:37:12.461+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:37:12.462+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:37:12.462+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:37:12.462+0000 7f12af924800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:37:12.462+0000 7f12af924800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 : miss 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:37:12.462+0000 7f12af924800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:37:12.463+0000 7f12af924800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:37:12.463+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:37:12.463+0000 7f12af924800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:37:12.463+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:37:12.463+0000 7f12af924800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 2023-12-03T11:37:12.586 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:37:12.463+0000 7f12aae7d640 10 monclient: get_auth_request con 0x556b01ca6290 auth_method 0 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:37:12.467+0000 7f12af924800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:37:12.467+0000 7f12af924800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:37:12.467+0000 7f12af924800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:37:12.467+0000 7f12af924800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25, ofs0, ns] 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725937 cookie 93918079364640 notifier 6689 bl.length()=893 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:37:12.467+0000 7f12917fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_sources= new_sources= 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential sources added=[] removed=[] 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_dests= new_dests= 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:37:12.468+0000 7f12af924800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential dests added=[] removed=[] 2023-12-03T11:37:12.587 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:37:12.469+0000 7f12af924800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:37:12.469+0000 7f12af924800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.0 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:37:12.472+0000 7f12af924800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f12af924800 time 2023-12-03T11:37:12.472362+0000 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f12b256eef8] 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:12.588 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:37:12.472+0000 7f12af924800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f12af924800 thread_name:radosgw-admin 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f12afe54db0] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f12afea154c] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f12b256ef52] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f12b256f0b6] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x556b00465a71] 2023-12-03T11:37:12.589 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x5b8) [0x556b004721d8] 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f12afe3feb0] 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:37:12.590 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:37:12.591 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:37:12.592 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:37:12.593 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:37:12.594 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:37:12.595 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11ddffb640 / http_manager 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11dffff640 / http_manager 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 7f11eaffd640 / ms_dispatch 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1270fe1640 / io_context_pool 2023-12-03T11:37:12.596 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12717e2640 / rgw_dt_lg_renew 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f128bfff640 / ms_dispatch 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12917fa640 / io_context_pool 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f1291ffb640 / io_context_pool 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12a8e79640 / ms_dispatch 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12aae7d640 / msgr-worker-2 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12ab67e640 / msgr-worker-1 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12abe7f640 / msgr-worker-0 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: 7f12af924800 / radosgw-admin 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:37:12.473706Z_b55bf184-1954-4512-930c-9943d066a3ea/log 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:37:12.597 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:37:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.520+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 20 realm 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.522+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.523+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.523+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.523+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.523+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.524+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.524+0000 7f84b751f800 20 searching for the correct realm 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.534+0000 7f84b314d640 0 -- 172.21.15.12:0/2479108201 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x558b63ba7bf0 msgr2=0x558b63ba9fe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.535+0000 7f84b751f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.535+0000 7f84b751f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:12.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.535+0000 7f84b751f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.535+0000 7f84b751f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.535+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 zone default found 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 4 Realm: () 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 zonegroup default 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.536+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.537+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.537+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.537+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.537+0000 7f84b751f800 20 started sync module instance, tier type = 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.537+0000 7f84b751f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.543+0000 7f84b394e640 0 -- 172.21.15.12:0/2479108201 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f84a8068aa0 msgr2=0x7f84a805e440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=0 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=1 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=2 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=3 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=4 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=5 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.546+0000 7f84b751f800 20 add_watcher() i=6 2023-12-03T11:37:12.605 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.547+0000 7f84b751f800 20 add_watcher() i=7 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.547+0000 7f84b751f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.548+0000 7f84797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.549+0000 7f84b751f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.549+0000 7f84b751f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f84b751f800 5 note: GC not initialized 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f83e7fff640 20 reqs_thread_entry: start 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f84b751f800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f84b751f800 20 Filter name: none 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f83e5ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.555+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:37:12.606 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.556+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.557+0000 7f84b751f800 20 rados_obj.operate() r=0 bl.length=413 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.557+0000 7f84b751f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.557+0000 7f84b751f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.557+0000 7f84b751f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.558+0000 7f84797e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.560+0000 7f84b751f800 20 remove_watcher() i=0 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.560+0000 7f84b751f800 2 removed watcher, disabling cache 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.561+0000 7f84b751f800 20 remove_watcher() i=1 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.562+0000 7f84b751f800 20 remove_watcher() i=2 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.563+0000 7f84b751f800 20 remove_watcher() i=3 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.564+0000 7f84b751f800 20 remove_watcher() i=4 2023-12-03T11:37:12.607 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.565+0000 7f84b751f800 20 remove_watcher() i=5 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.566+0000 7f84b751f800 20 remove_watcher() i=6 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:12.567+0000 7f84b751f800 20 remove_watcher() i=7 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:12.608 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:42.654 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-commit-target-layout --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:37:42.688 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:42.688 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:42.705 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.703+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.705+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.705+0000 7fa38140f800 20 realm 2023-12-03T11:37:42.707 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.705+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.706+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.706+0000 7fa38140f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.706+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.706+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.706+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.707+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:42.708 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.707+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.708+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:42.709 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.708+0000 7fa38140f800 20 searching for the correct realm 2023-12-03T11:37:42.721 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.720+0000 7fa37ca8c640 0 -- 172.21.15.12:0/4165330952 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55767486a5c0 msgr2=0x55767486ca10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:42.723 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.721+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 zone default found 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 4 Realm: () 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 zonegroup default 2023-12-03T11:37:42.724 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.722+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.723+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.723+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.723+0000 7fa38140f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.723+0000 7fa38140f800 20 started sync module instance, tier type = 2023-12-03T11:37:42.725 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.723+0000 7fa38140f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:42.732 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.731+0000 7fa37d28d640 0 -- 172.21.15.12:0/4165330952 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fa37005e330 msgr2=0x7fa37008a8b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:42.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=0 2023-12-03T11:37:42.734 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=1 2023-12-03T11:37:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=2 2023-12-03T11:37:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=3 2023-12-03T11:37:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=4 2023-12-03T11:37:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=5 2023-12-03T11:37:42.735 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.733+0000 7fa38140f800 20 add_watcher() i=6 2023-12-03T11:37:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.734+0000 7fa38140f800 20 add_watcher() i=7 2023-12-03T11:37:42.736 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.734+0000 7fa38140f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:42.737 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.736+0000 7fa342fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:42.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.737+0000 7fa38140f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:42.738 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.737+0000 7fa38140f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:42.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.743+0000 7fa38140f800 5 note: GC not initialized 2023-12-03T11:37:42.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.743+0000 7fa2b1ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:42.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.743+0000 7fa38140f800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:42.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.743+0000 7fa38140f800 20 Filter name: none 2023-12-03T11:37:42.745 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa2a37fe640 20 reqs_thread_entry: start 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 10 cache get: name=default.rgw.meta+root+abort-at-commit-target-layout : miss 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 10 cache put: name=default.rgw.meta+root+abort-at-commit-target-layout info.flags=0x11 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 10 adding default.rgw.meta+root+abort-at-commit-target-layout to cache LRU end 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 : miss 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.744+0000 7fa38140f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.746 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:42.747 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.745+0000 7fa38140f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 2023-12-03T11:37:42.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.749+0000 7fa38140f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:42.750 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.749+0000 7fa38140f800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:42.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.749+0000 7fa38140f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:42.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.749+0000 7fa38140f800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25, ofs0, ns] 2023-12-03T11:37:42.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa3637fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725939 cookie 93967249544224 notifier 6728 bl.length()=893 2023-12-03T11:37:42.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa3637fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:42.751 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa3637fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa3637fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_sources= new_sources= 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential sources added=[] removed=[] 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_dests= new_dests= 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.750+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential dests added=[] removed=[] 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.751+0000 7fa38140f800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.751+0000 7fa38140f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.0 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.751+0000 7fa38140f800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.751+0000 7fa38140f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.1.1 2023-12-03T11:37:42.752 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.751+0000 7fa38140f800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25.0 2023-12-03T11:37:42.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.753+0000 7fa38140f800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:42.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.753+0000 7fa38140f800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 2023-12-03T11:37:42.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa38140f800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:42.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa38140f800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa38140f800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa38140f800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25, ofs0, ns] 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa362ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725940 cookie 93967249544224 notifier 6728 bl.length()=861 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa362ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 info.flags=0x17 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa362ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-commit-target-layout:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 to cache LRU end 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.755+0000 7fa362ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.756+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_sources= new_sources= 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.756+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential sources added=[] removed=[] 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.756+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): orig_dests= new_dests= 2023-12-03T11:37:42.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.756+0000 7fa38140f800 20 handle_bi_update(): bucket=:abort-at-commit-target-layout[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25]): potential dests added=[] removed=[] 2023-12-03T11:37:42.758 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.757+0000 7fa38140f800 1 execute INFO: reshard of bucket "abort-at-commit-target-layout" completed successfully 2023-12-03T11:37:42.759 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.758+0000 7fa342fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:42.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.761+0000 7fa38140f800 20 remove_watcher() i=0 2023-12-03T11:37:42.762 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.761+0000 7fa38140f800 2 removed watcher, disabling cache 2023-12-03T11:37:42.763 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.762+0000 7fa38140f800 20 remove_watcher() i=1 2023-12-03T11:37:42.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.763+0000 7fa38140f800 20 remove_watcher() i=2 2023-12-03T11:37:42.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.764+0000 7fa38140f800 20 remove_watcher() i=3 2023-12-03T11:37:42.766 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.765+0000 7fa38140f800 20 remove_watcher() i=4 2023-12-03T11:37:42.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.766+0000 7fa38140f800 20 remove_watcher() i=5 2023-12-03T11:37:42.768 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.767+0000 7fa38140f800 20 remove_watcher() i=6 2023-12-03T11:37:42.769 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.768+0000 7fa38140f800 20 remove_watcher() i=7 2023-12-03T11:37:42.776 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-commit-target-layout 2023-12-03T11:37:42.804 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:42.804 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:42.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.817+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.820 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 realm 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:42.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:42.819+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.021+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.023 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.021+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.023+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.024 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.023+0000 7f785b09d800 20 searching for the correct realm 2023-12-03T11:37:43.035 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.034+0000 7f7858b9d640 0 -- 172.21.15.12:0/2789716804 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5591c6fecc50 msgr2=0x5591c6fef060 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.035+0000 7f785b09d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.035+0000 7f785b09d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.035+0000 7f785b09d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.035+0000 7f785b09d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.035+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.037 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 zone default found 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 4 Realm: () 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 zonegroup default 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.036+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 started sync module instance, tier type = 2023-12-03T11:37:43.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.037+0000 7f785b09d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.044+0000 7f785939e640 0 -- 172.21.15.12:0/2789716804 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5591c6ff4d30 msgr2=0x5591c6ff71c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=0 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=1 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=2 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=3 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=4 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=5 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.047+0000 7f785b09d800 20 add_watcher() i=6 2023-12-03T11:37:43.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.048+0000 7f785b09d800 20 add_watcher() i=7 2023-12-03T11:37:43.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.048+0000 7f785b09d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.051 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.050+0000 7f781efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.051+0000 7f785b09d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.051+0000 7f785b09d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:43.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f785b09d800 5 note: GC not initialized 2023-12-03T11:37:43.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f778dffb640 20 reqs_thread_entry: start 2023-12-03T11:37:43.058 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f785b09d800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f785b09d800 20 Filter name: none 2023-12-03T11:37:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f777f7fe640 20 reqs_thread_entry: start 2023-12-03T11:37:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.057+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.059 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.058+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.058+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.058+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.058+0000 7f785b09d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.058+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.059+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=197 2023-12-03T11:37:43.060 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.059+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.059+0000 7f785b09d800 20 rados_obj.operate() r=0 bl.length=381 2023-12-03T11:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.059+0000 7f785b09d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.060+0000 7f785b09d800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.061 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.060+0000 7f785b09d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.062 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.061+0000 7f781efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.063+0000 7f785b09d800 20 remove_watcher() i=0 2023-12-03T11:37:43.064 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.063+0000 7f785b09d800 2 removed watcher, disabling cache 2023-12-03T11:37:43.065 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.064+0000 7f785b09d800 20 remove_watcher() i=1 2023-12-03T11:37:43.066 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.065+0000 7f785b09d800 20 remove_watcher() i=2 2023-12-03T11:37:43.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.067+0000 7f785b09d800 20 remove_watcher() i=3 2023-12-03T11:37:43.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.068+0000 7f785b09d800 20 remove_watcher() i=4 2023-12-03T11:37:43.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.069+0000 7f785b09d800 20 remove_watcher() i=5 2023-12-03T11:37:43.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.070+0000 7f785b09d800 20 remove_watcher() i=6 2023-12-03T11:37:43.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.071+0000 7f785b09d800 20 remove_watcher() i=7 2023-12-03T11:37:43.077 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.077 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:43.077 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-commit-target-layout id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:37:43.077 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.162 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with EIO injected at do_reshard 2023-12-03T11:37:43.163 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.376 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:37:43.404 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:43.404 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:43.418 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.417+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 realm 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.420 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.419+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.420+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.421 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.420+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.421+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.421+0000 7f93ce540800 20 searching for the correct realm 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93cab04640 0 -- 172.21.15.12:0/933847721 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f93b40243e0 msgr2=0x7f93b4024820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93ce540800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93ce540800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93ce540800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93ce540800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.431+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.433 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.434 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 zone default found 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 4 Realm: () 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 zonegroup default 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.432+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.435 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 started sync module instance, tier type = 2023-12-03T11:37:43.436 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.433+0000 7f93ce540800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93cab04640 0 -- 172.21.15.12:0/933847721 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55bb1d36e140 msgr2=0x55bb1d370600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=0 2023-12-03T11:37:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=1 2023-12-03T11:37:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=2 2023-12-03T11:37:43.443 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=3 2023-12-03T11:37:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=4 2023-12-03T11:37:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=5 2023-12-03T11:37:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.442+0000 7f93ce540800 20 add_watcher() i=6 2023-12-03T11:37:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.443+0000 7f93ce540800 20 add_watcher() i=7 2023-12-03T11:37:43.444 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.443+0000 7f93ce540800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.445+0000 7f938ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.445+0000 7f93ce540800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.446 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.445+0000 7f93ce540800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.451+0000 7f93ce540800 5 note: GC not initialized 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.452+0000 7f92feffd640 20 reqs_thread_entry: start 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.452+0000 7f93ce540800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.452+0000 7f93ce540800 20 Filter name: none 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.452+0000 7f92fcff9640 20 reqs_thread_entry: start 2023-12-03T11:37:43.453 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.452+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.453+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.454 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.453+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.455 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.454+0000 7f93ce540800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.456 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.455+0000 7f93ce540800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.455+0000 7f93ce540800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.457 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.456+0000 7f938ffe7640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.458+0000 7f93ce540800 20 remove_watcher() i=0 2023-12-03T11:37:43.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.458+0000 7f93ce540800 2 removed watcher, disabling cache 2023-12-03T11:37:43.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.460+0000 7f93ce540800 20 remove_watcher() i=1 2023-12-03T11:37:43.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.461+0000 7f93ce540800 20 remove_watcher() i=2 2023-12-03T11:37:43.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.462+0000 7f93ce540800 20 remove_watcher() i=3 2023-12-03T11:37:43.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.463+0000 7f93ce540800 20 remove_watcher() i=4 2023-12-03T11:37:43.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.464+0000 7f93ce540800 20 remove_watcher() i=5 2023-12-03T11:37:43.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.465+0000 7f93ce540800 20 remove_watcher() i=6 2023-12-03T11:37:43.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.466+0000 7f93ce540800 20 remove_watcher() i=7 2023-12-03T11:37:43.471 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.471 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:43.471 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:43.472 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.473 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-error-at do_reshard 2023-12-03T11:37:43.505 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:43.506 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:43.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.519+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 realm 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:43.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.521+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.522+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.522+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.523+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.524 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.523+0000 7f8ddece4800 20 searching for the correct realm 2023-12-03T11:37:43.533 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.532+0000 7f8ddc7eb640 0 -- 172.21.15.12:0/3894386550 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x558892704a10 msgr2=0x558892706e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.535 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.534+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 zone default found 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 4 Realm: () 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 zonegroup default 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.537 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.535+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.536+0000 7f8ddece4800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.536+0000 7f8ddece4800 20 started sync module instance, tier type = 2023-12-03T11:37:43.538 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.536+0000 7f8ddece4800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.542 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.541+0000 7f8ddcfec640 0 -- 172.21.15.12:0/3894386550 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55889270c980 msgr2=0x55889270ee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=0 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=1 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=2 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=3 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=4 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=5 2023-12-03T11:37:43.547 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.545+0000 7f8ddece4800 20 add_watcher() i=6 2023-12-03T11:37:43.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.546+0000 7f8ddece4800 20 add_watcher() i=7 2023-12-03T11:37:43.548 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.546+0000 7f8ddece4800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.548+0000 7f8da27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.548+0000 7f8ddece4800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.549 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.548+0000 7f8ddece4800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 5 note: GC not initialized 2023-12-03T11:37:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8d117fa640 20 reqs_thread_entry: start 2023-12-03T11:37:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 20 Filter name: none 2023-12-03T11:37:43.558 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8d02ffd640 20 reqs_thread_entry: start 2023-12-03T11:37:43.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2023-12-03T11:37:43.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.559 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 : miss 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.557+0000 7f8ddece4800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.558+0000 7f8ddece4800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 2023-12-03T11:37:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8ddece4800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8ddece4800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8ddece4800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.563 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8ddece4800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26, ofs0, ns] 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8dc27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725975 cookie 94045060795360 notifier 6785 bl.length()=873 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8dc27fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.562+0000 7f8dc27fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.563+0000 7f8dc27fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.563+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_sources= new_sources= 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.563+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential sources added=[] removed=[] 2023-12-03T11:37:43.564 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.563+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_dests= new_dests= 2023-12-03T11:37:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.563+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential dests added=[] removed=[] 2023-12-03T11:37:43.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.564+0000 7f8ddece4800 -1 Injecting error=-5 at location=do_reshard 2023-12-03T11:37:43.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.566+0000 7f8ddece4800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:43.567 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.566+0000 7f8ddece4800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8ddece4800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8ddece4800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8ddece4800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8ddece4800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26, ofs0, ns] 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8dc27fc640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725976 cookie 94045060795360 notifier 6785 bl.length()=841 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8dc27fc640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8dc27fc640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.567+0000 7f8dc27fc640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.568+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_sources= new_sources= 2023-12-03T11:37:43.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.568+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential sources added=[] removed=[] 2023-12-03T11:37:43.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.568+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_dests= new_dests= 2023-12-03T11:37:43.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.568+0000 7f8ddece4800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential dests added=[] removed=[] 2023-12-03T11:37:43.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.568+0000 7f8ddece4800 1 execute INFO: reshard of bucket "error-at-do-reshard" canceled due to errors 2023-12-03T11:37:43.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.570+0000 7f8da27e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.571+0000 7f8ddece4800 20 remove_watcher() i=0 2023-12-03T11:37:43.573 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.571+0000 7f8ddece4800 2 removed watcher, disabling cache 2023-12-03T11:37:43.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.572+0000 7f8ddece4800 20 remove_watcher() i=1 2023-12-03T11:37:43.574 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.573+0000 7f8ddece4800 20 remove_watcher() i=2 2023-12-03T11:37:43.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.575+0000 7f8ddece4800 20 remove_watcher() i=3 2023-12-03T11:37:43.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.576+0000 7f8ddece4800 20 remove_watcher() i=4 2023-12-03T11:37:43.578 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.577+0000 7f8ddece4800 20 remove_watcher() i=5 2023-12-03T11:37:43.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.577+0000 7f8ddece4800 20 remove_watcher() i=6 2023-12-03T11:37:43.579 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.578+0000 7f8ddece4800 20 remove_watcher() i=7 2023-12-03T11:37:43.586 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:37:43.614 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:43.614 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:43.628 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.627+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.630 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 realm 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.629+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.630+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.630+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.631+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.632 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.631+0000 7fb34320f800 20 searching for the correct realm 2023-12-03T11:37:43.642 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.640+0000 7fb33e947640 0 -- 172.21.15.12:0/4181719555 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55ef4267a6f0 msgr2=0x55ef4267cae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.642+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 zone default found 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 4 Realm: () 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 zonegroup default 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.643+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.644+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.644+0000 7fb34320f800 20 started sync module instance, tier type = 2023-12-03T11:37:43.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.644+0000 7fb34320f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.650+0000 7fb33f148640 0 -- 172.21.15.12:0/4181719555 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55ef426827f0 msgr2=0x55ef42684cb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=0 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=1 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=2 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=3 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=4 2023-12-03T11:37:43.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.653+0000 7fb34320f800 20 add_watcher() i=5 2023-12-03T11:37:43.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.654+0000 7fb34320f800 20 add_watcher() i=6 2023-12-03T11:37:43.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.654+0000 7fb34320f800 20 add_watcher() i=7 2023-12-03T11:37:43.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.654+0000 7fb34320f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.655+0000 7fb27ffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.656+0000 7fb34320f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.657 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.656+0000 7fb34320f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:43.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb34320f800 5 note: GC not initialized 2023-12-03T11:37:43.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb26effd640 20 reqs_thread_entry: start 2023-12-03T11:37:43.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb34320f800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:43.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb34320f800 20 Filter name: none 2023-12-03T11:37:43.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb26cff9640 20 reqs_thread_entry: start 2023-12-03T11:37:43.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.662+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.663+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.664+0000 7fb34320f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.664+0000 7fb34320f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:43.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.664+0000 7fb34320f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.665+0000 7fb34320f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.666 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.665+0000 7fb27ffff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.668+0000 7fb34320f800 20 remove_watcher() i=0 2023-12-03T11:37:43.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.668+0000 7fb34320f800 2 removed watcher, disabling cache 2023-12-03T11:37:43.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.669+0000 7fb34320f800 20 remove_watcher() i=1 2023-12-03T11:37:43.672 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.672+0000 7fb34320f800 20 remove_watcher() i=2 2023-12-03T11:37:43.674 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.673+0000 7fb34320f800 20 remove_watcher() i=3 2023-12-03T11:37:43.675 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.674+0000 7fb34320f800 20 remove_watcher() i=4 2023-12-03T11:37:43.676 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.675+0000 7fb34320f800 20 remove_watcher() i=5 2023-12-03T11:37:43.677 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.676+0000 7fb34320f800 20 remove_watcher() i=6 2023-12-03T11:37:43.678 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.677+0000 7fb34320f800 20 remove_watcher() i=7 2023-12-03T11:37:43.683 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.683 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:43.683 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:43.683 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:43.736 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket error-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:37:43.766 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:43.766 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:43.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.780+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 realm 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.782+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.783+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.783+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.784+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.784+0000 7fb7df926800 20 searching for the correct realm 2023-12-03T11:37:43.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.793+0000 7fb7db59c640 0 -- 172.21.15.12:0/1886951010 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55fda77b8f70 msgr2=0x55fda77bb3c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.795+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 zone default found 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 4 Realm: () 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 zonegroup default 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.798 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 started sync module instance, tier type = 2023-12-03T11:37:43.799 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.796+0000 7fb7df926800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.803+0000 7fb7dbd9d640 0 -- 172.21.15.12:0/1886951010 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55fda77c10b0 msgr2=0x55fda77c3570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=0 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=1 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=2 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=3 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=4 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=5 2023-12-03T11:37:43.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.806+0000 7fb7df926800 20 add_watcher() i=6 2023-12-03T11:37:43.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.807+0000 7fb7df926800 20 add_watcher() i=7 2023-12-03T11:37:43.808 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.807+0000 7fb7df926800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.808+0000 7fb7a17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.808+0000 7fb7df926800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.808+0000 7fb7df926800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.814+0000 7fb7df926800 5 note: GC not initialized 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb70ffff640 20 reqs_thread_entry: start 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 20 Filter name: none 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb70dffb640 20 reqs_thread_entry: start 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 10 cache get: name=default.rgw.meta+root+error-at-do-reshard : miss 2023-12-03T11:37:43.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 10 cache put: name=default.rgw.meta+root+error-at-do-reshard info.flags=0x11 2023-12-03T11:37:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 10 adding default.rgw.meta+root+error-at-do-reshard to cache LRU end 2023-12-03T11:37:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 : miss 2023-12-03T11:37:43.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.815+0000 7fb7df926800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 10 adding default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.818 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.816+0000 7fb7df926800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 2023-12-03T11:37:43.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.820+0000 7fb7df926800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.820+0000 7fb7df926800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.821 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.820+0000 7fb7df926800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.820+0000 7fb7df926800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26, ofs0, ns] 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7c17fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725977 cookie 94547925031808 notifier 6824 bl.length()=873 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7c17fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7c17fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7c17fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_sources= new_sources= 2023-12-03T11:37:43.822 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential sources added=[] removed=[] 2023-12-03T11:37:43.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_dests= new_dests= 2023-12-03T11:37:43.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.821+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential dests added=[] removed=[] 2023-12-03T11:37:43.823 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.822+0000 7fb7df926800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26.1.0 2023-12-03T11:37:43.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.822+0000 7fb7df926800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26.1.0 2023-12-03T11:37:43.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.822+0000 7fb7df926800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26.1.1 2023-12-03T11:37:43.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.822+0000 7fb7df926800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26.1.1 2023-12-03T11:37:43.824 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.822+0000 7fb7df926800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26.0 2023-12-03T11:37:43.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.825+0000 7fb7df926800 10 cache get: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 : hit (requested=0x17, cached=0x17) 2023-12-03T11:37:43.826 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.825+0000 7fb7df926800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 10 cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 10 moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 10 distributing notification oid=default.rgw.control:notify.2 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26, ofs0, ns] 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7c1ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725978 cookie 94547925031808 notifier 6824 bl.length()=841 2023-12-03T11:37:43.827 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7c1ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 info.flags=0x17 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7c1ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.error-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 to cache LRU end 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7c1ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_sources= new_sources= 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential sources added=[] removed=[] 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): orig_dests= new_dests= 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.826+0000 7fb7df926800 20 handle_bi_update(): bucket=:error-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26]): potential dests added=[] removed=[] 2023-12-03T11:37:43.828 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.827+0000 7fb7df926800 1 execute INFO: reshard of bucket "error-at-do-reshard" completed successfully 2023-12-03T11:37:43.830 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.829+0000 7fb7a17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.832+0000 7fb7df926800 20 remove_watcher() i=0 2023-12-03T11:37:43.833 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.832+0000 7fb7df926800 2 removed watcher, disabling cache 2023-12-03T11:37:43.834 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.833+0000 7fb7df926800 20 remove_watcher() i=1 2023-12-03T11:37:43.835 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.834+0000 7fb7df926800 20 remove_watcher() i=2 2023-12-03T11:37:43.836 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.835+0000 7fb7df926800 20 remove_watcher() i=3 2023-12-03T11:37:43.837 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.836+0000 7fb7df926800 20 remove_watcher() i=4 2023-12-03T11:37:43.838 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.837+0000 7fb7df926800 20 remove_watcher() i=5 2023-12-03T11:37:43.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.838+0000 7fb7df926800 20 remove_watcher() i=6 2023-12-03T11:37:43.840 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.839+0000 7fb7df926800 20 remove_watcher() i=7 2023-12-03T11:37:43.847 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket error-at-do-reshard 2023-12-03T11:37:43.874 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:43.875 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:43.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.888+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 20 realm 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.889+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.890+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.890+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.891+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.891+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.891+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:43.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.891+0000 7f5a47537800 20 searching for the correct realm 2023-12-03T11:37:43.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.901+0000 7f5a432cd640 0 -- 172.21.15.12:0/3845615429 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x556edc290b60 msgr2=0x556edc292f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.902+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 zone default found 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 4 Realm: () 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:43.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 zonegroup default 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.903+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.904+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.904+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.904+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.904+0000 7f5a47537800 20 started sync module instance, tier type = 2023-12-03T11:37:43.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.904+0000 7f5a47537800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:43.912 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.911+0000 7f5a43ace640 0 -- 172.21.15.12:0/3845615429 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x556edc298c60 msgr2=0x556edc29b120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:43.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=0 2023-12-03T11:37:43.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=1 2023-12-03T11:37:43.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=2 2023-12-03T11:37:43.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=3 2023-12-03T11:37:43.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=4 2023-12-03T11:37:43.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=5 2023-12-03T11:37:43.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.914+0000 7f5a47537800 20 add_watcher() i=6 2023-12-03T11:37:43.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.915+0000 7f5a47537800 20 add_watcher() i=7 2023-12-03T11:37:43.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.915+0000 7f5a47537800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:43.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.916+0000 7f5a097e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:43.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.917+0000 7f5a47537800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:43.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:43.917+0000 7f5a47537800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.119+0000 7f5a432cd640 0 --2- 172.21.15.12:0/526809180 >> [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] conn(0x556edc2d4120 0x556edc2cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:37:44.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5a47537800 5 note: GC not initialized 2023-12-03T11:37:44.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5977fff640 20 reqs_thread_entry: start 2023-12-03T11:37:44.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5a47537800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5a47537800 20 Filter name: none 2023-12-03T11:37:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5975ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.124+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.125+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.125+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.125+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.125+0000 7f5a47537800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.125+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.126+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.126+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.126+0000 7f5a47537800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.126+0000 7f5a47537800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.127+0000 7f5a47537800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.127+0000 7f5a47537800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.128+0000 7f5a097e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.130+0000 7f5a47537800 20 remove_watcher() i=0 2023-12-03T11:37:44.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.130+0000 7f5a47537800 2 removed watcher, disabling cache 2023-12-03T11:37:44.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.131+0000 7f5a47537800 20 remove_watcher() i=1 2023-12-03T11:37:44.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.132+0000 7f5a47537800 20 remove_watcher() i=2 2023-12-03T11:37:44.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.133+0000 7f5a47537800 20 remove_watcher() i=3 2023-12-03T11:37:44.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.134+0000 7f5a47537800 20 remove_watcher() i=4 2023-12-03T11:37:44.136 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.135+0000 7f5a47537800 20 remove_watcher() i=5 2023-12-03T11:37:44.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.136+0000 7f5a47537800 20 remove_watcher() i=6 2023-12-03T11:37:44.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.137+0000 7f5a47537800 20 remove_watcher() i=7 2023-12-03T11:37:44.143 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.143 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:44.143 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: error-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:37:44.143 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.230 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: reshard bucket with abort at do_reshard 2023-12-03T11:37:44.230 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.443 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:37:44.470 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:44.471 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:44.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.483+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 realm 2023-12-03T11:37:44.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:44.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.485+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.486+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.486+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.487+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:44.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.487+0000 7f683022f800 20 searching for the correct realm 2023-12-03T11:37:44.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.496+0000 7f682b4fa640 0 -- 172.21.15.12:0/411097151 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5570ec2fb190 msgr2=0x5570ec2fd580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.498+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 zone default found 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 4 Realm: () 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 zonegroup default 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.499+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.500+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.500+0000 7f683022f800 20 started sync module instance, tier type = 2023-12-03T11:37:44.502 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.500+0000 7f683022f800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:44.507 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.506+0000 7f682bcfb640 0 -- 172.21.15.12:0/411097151 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5570ec303070 msgr2=0x5570ec305500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=0 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=1 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=2 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=3 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=4 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=5 2023-12-03T11:37:44.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.508+0000 7f683022f800 20 add_watcher() i=6 2023-12-03T11:37:44.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.509+0000 7f683022f800 20 add_watcher() i=7 2023-12-03T11:37:44.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.509+0000 7f683022f800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:44.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.511+0000 7f67f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.511+0000 7f683022f800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:44.512 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.511+0000 7f683022f800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.517+0000 7f683022f800 5 note: GC not initialized 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.517+0000 7f675ffff640 20 reqs_thread_entry: start 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.517+0000 7f683022f800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.518+0000 7f683022f800 20 Filter name: none 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.518+0000 7f675dffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.518+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.518+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.518+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.519+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.519+0000 7f683022f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.519+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.519+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.519+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.520+0000 7f683022f800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.520+0000 7f683022f800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.521 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.520+0000 7f683022f800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.522 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.521+0000 7f683022f800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.523 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.522+0000 7f67f17e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.524+0000 7f683022f800 20 remove_watcher() i=0 2023-12-03T11:37:44.525 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.524+0000 7f683022f800 2 removed watcher, disabling cache 2023-12-03T11:37:44.526 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.525+0000 7f683022f800 20 remove_watcher() i=1 2023-12-03T11:37:44.527 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.526+0000 7f683022f800 20 remove_watcher() i=2 2023-12-03T11:37:44.528 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.527+0000 7f683022f800 20 remove_watcher() i=3 2023-12-03T11:37:44.529 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.528+0000 7f683022f800 20 remove_watcher() i=4 2023-12-03T11:37:44.530 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.529+0000 7f683022f800 20 remove_watcher() i=5 2023-12-03T11:37:44.531 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.530+0000 7f683022f800 20 remove_watcher() i=6 2023-12-03T11:37:44.532 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.531+0000 7f683022f800 20 remove_watcher() i=7 2023-12-03T11:37:44.537 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.537 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:44.537 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:44.537 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.538 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 --inject-abort-at do_reshard 2023-12-03T11:37:44.572 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:44.572 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:44.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.585+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 realm 2023-12-03T11:37:44.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.589 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.589+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:44.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.589+0000 7f7e50d42800 20 searching for the correct realm 2023-12-03T11:37:44.599 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.598+0000 7f7e4cb03640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x562f4345d340 msgr2=0x562f4345f730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.602 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zone default found 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Realm: () 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zonegroup default 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.603 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started sync module instance, tier type = 2023-12-03T11:37:44.604 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:44.608 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.607+0000 7f7e4d304640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x562f43465160 msgr2=0x562f43467620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:44.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=0 2023-12-03T11:37:44.611 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=1 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=2 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=3 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=4 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=5 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=6 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.611+0000 7f7e50d42800 20 add_watcher() i=7 2023-12-03T11:37:44.612 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.611+0000 7f7e50d42800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:44.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.613+0000 7f7e12fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:44.614 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7e50d42800 5 note: GC not initialized 2023-12-03T11:37:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7d81ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7e50d42800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7e50d42800 20 Filter name: none 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7d737fe640 20 reqs_thread_entry: start 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.619+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 : miss 2023-12-03T11:37:44.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.620+0000 7f7e50d42800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 2023-12-03T11:37:44.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e50d42800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.626 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e50d42800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27, ofs0, ns] 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725963 cookie 94760992107232 notifier 7251 bl.length()=873 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_sources= new_sources= 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential sources added=[] removed=[] 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_dests= new_dests= 2023-12-03T11:37:44.627 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential dests added=[] removed=[] 2023-12-03T11:37:44.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7e50d42800 time 2023-12-03T11:37:44.628187+0000 2023-12-03T11:37:44.628 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7e5396eef8] 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr:*** Caught signal (Aborted) ** 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f7e50d42800 thread_name:radosgw-admin 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.627+0000 7f7e50d42800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7e50d42800 time 2023-12-03T11:37:44.628187+0000 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:44.629 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7e5396eef8] 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:44.630 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7e51254db0] 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f7e512a154c] 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:44.650 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7e5396ef52] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.628+0000 7f7e50d42800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f7e50d42800 thread_name:radosgw-admin 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7e51254db0] 2023-12-03T11:37:44.651 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f7e512a154c] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7e5396ef52] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:37:44.579+0000 7f7e4a2fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.652 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:37:44.579+0000 7f7e4a2fe640 10 monclient: dump: 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:37:44.579+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7245 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:37:44.579+0000 7f7e4b300640 4 set_mon_vals no callback set 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.653 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:37:44.579+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: get_monmap_and_config 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:37:44.580+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.654 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.655 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:37:44.580+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.656 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:37:44.580+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:37:44.581+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:37:44.581+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2dcac4e8 EEXIST 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4343bb60 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f43446bd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f43447030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.657 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.764089s 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f43447030 auth_method 0 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f43446bd0 auth_method 0 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f4343bb60 auth_method 0 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.658 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_done global_id 7248 payload 274 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:37:44.581+0000 7f7e4c302640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.659 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:37:44.582+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7e38069820 auth_method 0 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: dump: 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.660 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7248 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:37:44.582+0000 7f7e50d42800 4 set_mon_vals no callback set 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: dump: 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.661 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: shutdown 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:37:44.582+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.662 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.663 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: starting msgr at 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: starting objecter 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command objecter_requests cmddesc objecter_requests hook 0x562f432c4b20 EEXIST 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: setting wanted keys 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: calling monclient init 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.664 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x562f4340b1a0 EEXIST 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4343bb60 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f43446bd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f43447030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.665 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.766421s 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f43446bd0 auth_method 0 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f43447030 auth_method 0 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.666 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f4343bb60 auth_method 0 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_done global_id 7251 payload 824 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:37:44.584+0000 7f7e4c302640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: dump: 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.667 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: dump: 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.668 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:37:44.584+0000 7f7e337fe640 4 set_mon_vals no callback set 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:37:44.584+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7251 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:37:44.584+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:37:44.585+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.669 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:37:44.586+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e28069820 auth_method 0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:37:44.586+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f43459120 auth_method 0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 realm 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:37:44.587+0000 7f7e50d42800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:37:44.588+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x562f4345d340 auth_method 0 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.670 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:37:44.588+0000 7f7e4c302640 10 monclient: get_auth_request con 0x562f43460980 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:37:44.589+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:37:44.589+0000 7f7e50d42800 20 searching for the correct realm 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:37:44.589+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f43465160 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:37:44.590+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e40076070 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:37:44.591+0000 7f7e4c302640 10 monclient: get_auth_request con 0x7f7e3c0090e0 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:37:44.593+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7e400093c0 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:37:44.595+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e400a2c60 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:37:44.598+0000 7f7e4cb03640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x562f4345d340 msgr2=0x562f4345f730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 1 client.7251.objecter ms_handle_reset 0x562f4345d340 session 0x562f431cb320 osd.5 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 10 monclient: _renew_subs 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:37:44.599+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e28079d00 auth_method 0 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:44.671 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zone default found 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Realm: () 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:37:44.601+0000 7f7e50d42800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zonegroup default 2023-12-03T11:37:44.672 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started sync module instance, tier type = 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:37:44.607+0000 7f7e4d304640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x562f43465160 msgr2=0x562f43467620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:37:44.607+0000 7f7e31ffb640 1 client.7251.objecter ms_handle_reset 0x562f43465160 session 0x562f43464ae0 osd.3 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:37:44.607+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f4345d340 auth_method 0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=0 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=1 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=2 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=3 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=4 2023-12-03T11:37:44.673 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=5 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=6 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:37:44.611+0000 7f7e50d42800 20 add_watcher() i=7 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:37:44.611+0000 7f7e50d42800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:37:44.613+0000 7f7e12fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.674 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:37:44.613+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache list hook 0x562f43401ef0 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache inspect hook 0x562f43401ef0 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache erase hook 0x562f43401ef0 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache zap hook 0x562f43401ef0 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: get_monmap_and_config 2023-12-03T11:37:44.675 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:37:44.613+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.676 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:37:44.613+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.677 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:37:44.614+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.678 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:37:44.614+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:37:44.614+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2dcaca68 EEXIST 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f434a07d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f4349f7e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4349b4b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.796984s 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f434a07d0 auth_method 0 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.679 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f4349f7e0 auth_method 0 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f4349b4b0 auth_method 0 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.680 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_done global_id 7254 payload 274 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: dump: 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.681 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7254 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: dump: 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.682 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:37:44.615+0000 7f7e50d42800 4 set_mon_vals no callback set 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: shutdown 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:37:44.615+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.683 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.684 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: starting msgr at 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: starting objecter 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command objecter_requests cmddesc objecter_requests hook 0x562f43460430 EEXIST 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: setting wanted keys 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: calling monclient init 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.685 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.686 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x562f434a15c0 EEXIST 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f4349b4b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f434a07d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f434b3700 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.799485s 2023-12-03T11:37:44.687 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f4349b4b0 auth_method 0 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f434a07d0 auth_method 0 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f434b3700 auth_method 0 2023-12-03T11:37:44.688 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_done global_id 7257 payload 824 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: dump: 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.689 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: dump: 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.690 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:37:44.617+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7257 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:37:44.617+0000 7f7e127e4640 4 set_mon_vals no callback set 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:37:44.617+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:37:44.617+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cr dump hook 0x562f434a6c18 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.691 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:37:44.619+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7d88069820 auth_method 0 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 note: GC not initialized 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:37:44.619+0000 7f7d81ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace show hook 0x7f7e40004460 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace history hook 0x7f7e40004460 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace active hook 0x7f7e40004460 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace active_short hook 0x7f7e40004460 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 Filter name: none 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:37:44.619+0000 7f7d737fe640 20 reqs_thread_entry: start 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:37:44.619+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.692 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 : miss 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:37:44.621+0000 7f7e4c302640 10 monclient: get_auth_request con 0x562f434ef3f0 auth_method 0 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27, ofs0, ns] 2023-12-03T11:37:44.693 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725963 cookie 94760992107232 notifier 7251 bl.length()=873 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_sources= new_sources= 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential sources added=[] removed=[] 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_dests= new_dests= 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential dests added=[] removed=[] 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:37:44.627+0000 7f7e50d42800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7e50d42800 time 2023-12-03T11:37:44.628187+0000 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7e5396eef8] 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.694 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:37:44.628+0000 7f7e50d42800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f7e50d42800 thread_name:radosgw-admin 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7e51254db0] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f7e512a154c] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7e5396ef52] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.695 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:37:44.696 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:37:44.697 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:37:44.698 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:37:44.699 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:37:44.700 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:37:44.701 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7d737fe640 / http_manager 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7d81ffb640 / http_manager 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e10fe1640 / ms_dispatch 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e127e4640 / io_context_pool 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e12fe5640 / rgw_dt_lg_renew 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e31ffb640 / ms_dispatch 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e337fe640 / io_context_pool 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4a2fe640 / ms_dispatch 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4b300640 / io_context_pool 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4c302640 / msgr-worker-2 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4cb03640 / msgr-worker-1 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4d304640 / msgr-worker-0 2023-12-03T11:37:44.702 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e50d42800 / radosgw-admin 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/log/ceph/ceph-client.admin.104309.log 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:--- begin dump of recent events --- 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: -499> 2023-12-03T11:37:44.579+0000 7f7e4a2fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: -498> 2023-12-03T11:37:44.579+0000 7f7e4a2fe640 10 monclient: dump: 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.703 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -497> 2023-12-03T11:37:44.579+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7245 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -496> 2023-12-03T11:37:44.579+0000 7f7e4b300640 4 set_mon_vals no callback set 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -495> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -494> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -493> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -492> 2023-12-03T11:37:44.579+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -491> 2023-12-03T11:37:44.579+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -490> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: get_monmap_and_config 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -489> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -488> 2023-12-03T11:37:44.580+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr: -487> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.704 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -486> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -485> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -484> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding auth protocol: cephx 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -483> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -482> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -481> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -480> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -479> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -478> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.705 INFO:tasks.workunit.client.0.smithi012.stderr: -477> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -476> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -475> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -474> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -473> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: crc 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -472> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x562f4340cd10) adding con mode: secure 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -471> 2023-12-03T11:37:44.580+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -470> 2023-12-03T11:37:44.580+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -469> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -468> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -467> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding auth protocol: cephx 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -466> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -465> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -464> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -463> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.706 INFO:tasks.workunit.client.0.smithi012.stderr: -462> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -461> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -460> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -459> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -458> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -457> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -456> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: crc 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -455> 2023-12-03T11:37:44.580+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac3a0) adding con mode: secure 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -454> 2023-12-03T11:37:44.580+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -453> 2023-12-03T11:37:44.581+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -452> 2023-12-03T11:37:44.581+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2dcac4e8 EEXIST 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -451> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -450> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient: _add_conns ranks=[1,2,0] 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -449> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4343bb60 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -448> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f43446bd0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -447> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f43447030 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.707 INFO:tasks.workunit.client.0.smithi012.stderr: -446> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -445> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -444> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -443> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -442> 2023-12-03T11:37:44.581+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.764089s 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -441> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f43447030 auth_method 0 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -440> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -439> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -438> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -437> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f43446bd0 auth_method 0 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -436> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -435> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -434> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -433> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f4343bb60 auth_method 0 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -432> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.708 INFO:tasks.workunit.client.0.smithi012.stderr: -431> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -430> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -429> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -428> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -427> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -426> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -425> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -424> 2023-12-03T11:37:44.581+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -423> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -422> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -421> 2023-12-03T11:37:44.581+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -420> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient(hunting): handle_auth_done global_id 7248 payload 274 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -419> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -418> 2023-12-03T11:37:44.581+0000 7f7e4c302640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -417> 2023-12-03T11:37:44.581+0000 7f7e4c302640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.709 INFO:tasks.workunit.client.0.smithi012.stderr: -416> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -415> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -414> 2023-12-03T11:37:44.581+0000 7f7e4a2fe640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -413> 2023-12-03T11:37:44.582+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7e38069820 auth_method 0 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -412> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -411> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr: -410> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: dump: 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.710 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -409> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -408> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -407> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -406> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7248 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -405> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -404> 2023-12-03T11:37:44.582+0000 7f7e50d42800 4 set_mon_vals no callback set 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -403> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -402> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr: -401> 2023-12-03T11:37:44.582+0000 7f7e337fe640 10 monclient: dump: 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.711 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr: -400> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: shutdown 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr: -399> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr: -398> 2023-12-03T11:37:44.582+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr: -397> 2023-12-03T11:37:44.582+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.712 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -396> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -395> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -394> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding auth protocol: cephx 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -393> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -392> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -391> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -390> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -389> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -388> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -387> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -386> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.713 INFO:tasks.workunit.client.0.smithi012.stderr: -385> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -384> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -383> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: crc 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -382> 2023-12-03T11:37:44.582+0000 7f7e50d42800 5 AuthRegistry(0x562f4343e330) adding con mode: secure 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -381> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -380> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: starting msgr at 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -379> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: starting objecter 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -378> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command objecter_requests cmddesc objecter_requests hook 0x562f432c4b20 EEXIST 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -377> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: setting wanted keys 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -376> 2023-12-03T11:37:44.583+0000 7f7e50d42800 1 librados: calling monclient init 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -375> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -374> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -373> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -372> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding auth protocol: cephx 2023-12-03T11:37:44.714 INFO:tasks.workunit.client.0.smithi012.stderr: -371> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -370> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -369> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -368> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -367> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -366> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -365> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -364> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -363> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -362> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -361> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: crc 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -360> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 AuthRegistry(0x562f4340b058) adding con mode: secure 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -359> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -358> 2023-12-03T11:37:44.583+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -357> 2023-12-03T11:37:44.583+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x562f4340b1a0 EEXIST 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -356> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.715 INFO:tasks.workunit.client.0.smithi012.stderr: -355> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient: _add_conns ranks=[1,0,2] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -354> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4343bb60 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -353> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f43446bd0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -352> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f43447030 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -351> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -350> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -349> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -348> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -347> 2023-12-03T11:37:44.583+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.766421s 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -346> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f43446bd0 auth_method 0 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -345> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -344> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -343> 2023-12-03T11:37:44.583+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -342> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f43447030 auth_method 0 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -341> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.716 INFO:tasks.workunit.client.0.smithi012.stderr: -340> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -339> 2023-12-03T11:37:44.583+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -338> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -337> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -336> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -335> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -334> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -333> 2023-12-03T11:37:44.584+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -332> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f4343bb60 auth_method 0 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -331> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -330> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -329> 2023-12-03T11:37:44.584+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -328> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient(hunting): handle_auth_done global_id 7251 payload 824 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -327> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -326> 2023-12-03T11:37:44.584+0000 7f7e4c302640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.717 INFO:tasks.workunit.client.0.smithi012.stderr: -325> 2023-12-03T11:37:44.584+0000 7f7e4c302640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: -324> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: -323> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: -322> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: dump: 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: -321> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.718 INFO:tasks.workunit.client.0.smithi012.stderr: -320> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: -319> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: -318> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: -317> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: -316> 2023-12-03T11:37:44.584+0000 7f7e31ffb640 10 monclient: dump: 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.719 INFO:tasks.workunit.client.0.smithi012.stderr: -315> 2023-12-03T11:37:44.584+0000 7f7e337fe640 4 set_mon_vals no callback set 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -314> 2023-12-03T11:37:44.584+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7251 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -313> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -312> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -311> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -310> 2023-12-03T11:37:44.584+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -309> 2023-12-03T11:37:44.584+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -308> 2023-12-03T11:37:44.585+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -307> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -306> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -305> 2023-12-03T11:37:44.586+0000 7f7e31ffb640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -304> 2023-12-03T11:37:44.586+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e28069820 auth_method 0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -303> 2023-12-03T11:37:44.586+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f43459120 auth_method 0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -302> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -301> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 realm 2023-12-03T11:37:44.720 INFO:tasks.workunit.client.0.smithi012.stderr: -300> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -299> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -298> 2023-12-03T11:37:44.587+0000 7f7e50d42800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -297> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -296> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -295> 2023-12-03T11:37:44.587+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -294> 2023-12-03T11:37:44.588+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x562f4345d340 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -293> 2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -292> 2023-12-03T11:37:44.588+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -291> 2023-12-03T11:37:44.588+0000 7f7e4c302640 10 monclient: get_auth_request con 0x562f43460980 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -290> 2023-12-03T11:37:44.589+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -289> 2023-12-03T11:37:44.589+0000 7f7e50d42800 20 searching for the correct realm 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -288> 2023-12-03T11:37:44.589+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f43465160 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -287> 2023-12-03T11:37:44.590+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e40076070 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -286> 2023-12-03T11:37:44.591+0000 7f7e4c302640 10 monclient: get_auth_request con 0x7f7e3c0090e0 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -285> 2023-12-03T11:37:44.593+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7e400093c0 auth_method 0 2023-12-03T11:37:44.721 INFO:tasks.workunit.client.0.smithi012.stderr: -284> 2023-12-03T11:37:44.595+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e400a2c60 auth_method 0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -283> 2023-12-03T11:37:44.598+0000 7f7e4cb03640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x562f4345d340 msgr2=0x562f4345f730 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -282> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 1 client.7251.objecter ms_handle_reset 0x562f4345d340 session 0x562f431cb320 osd.5 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -281> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 10 monclient: _renew_subs 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -280> 2023-12-03T11:37:44.598+0000 7f7e31ffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -279> 2023-12-03T11:37:44.599+0000 7f7e4cb03640 10 monclient: get_auth_request con 0x7f7e28079d00 auth_method 0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -278> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -277> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -276> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -275> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -274> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -273> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -272> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -271> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -270> 2023-12-03T11:37:44.600+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -269> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:44.722 INFO:tasks.workunit.client.0.smithi012.stderr: -268> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zone default found 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -267> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Realm: () 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -266> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -265> 2023-12-03T11:37:44.601+0000 7f7e50d42800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -264> 2023-12-03T11:37:44.601+0000 7f7e50d42800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -263> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 zonegroup default 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -262> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -261> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -260> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -259> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -258> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -257> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -256> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started sync module instance, tier type = 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -255> 2023-12-03T11:37:44.601+0000 7f7e50d42800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -254> 2023-12-03T11:37:44.607+0000 7f7e4d304640 0 -- 172.21.15.12:0/136278292 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x562f43465160 msgr2=0x562f43467620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -253> 2023-12-03T11:37:44.607+0000 7f7e31ffb640 1 client.7251.objecter ms_handle_reset 0x562f43465160 session 0x562f43464ae0 osd.3 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -252> 2023-12-03T11:37:44.607+0000 7f7e4d304640 10 monclient: get_auth_request con 0x562f4345d340 auth_method 0 2023-12-03T11:37:44.723 INFO:tasks.workunit.client.0.smithi012.stderr: -251> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=0 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -250> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=1 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -249> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=2 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -248> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=3 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -247> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=4 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -246> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=5 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -245> 2023-12-03T11:37:44.610+0000 7f7e50d42800 20 add_watcher() i=6 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -244> 2023-12-03T11:37:44.611+0000 7f7e50d42800 20 add_watcher() i=7 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -243> 2023-12-03T11:37:44.611+0000 7f7e50d42800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -242> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -241> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -240> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding auth protocol: cephx 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -239> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -238> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -237> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -236> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -235> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -234> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.724 INFO:tasks.workunit.client.0.smithi012.stderr: -233> 2023-12-03T11:37:44.613+0000 7f7e12fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -232> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -231> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -230> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -229> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -228> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: crc 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -227> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcacdd0) adding con mode: secure 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -226> 2023-12-03T11:37:44.613+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -225> 2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -224> 2023-12-03T11:37:44.613+0000 7f7e50d42800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -223> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache list hook 0x562f43401ef0 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -222> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache inspect hook 0x562f43401ef0 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -221> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache erase hook 0x562f43401ef0 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -220> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cache zap hook 0x562f43401ef0 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -219> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: get_monmap_and_config 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -218> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.725 INFO:tasks.workunit.client.0.smithi012.stderr: -217> 2023-12-03T11:37:44.613+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: -216> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: -215> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: -214> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: -213> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding auth protocol: cephx 2023-12-03T11:37:44.726 INFO:tasks.workunit.client.0.smithi012.stderr: -212> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -211> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -210> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -209> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -208> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -207> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -206> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -205> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -204> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -203> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -202> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: crc 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -201> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x562f4349df60) adding con mode: secure 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -200> 2023-12-03T11:37:44.613+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -199> 2023-12-03T11:37:44.613+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -198> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -197> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -196> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding auth protocol: cephx 2023-12-03T11:37:44.727 INFO:tasks.workunit.client.0.smithi012.stderr: -195> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -194> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -193> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -192> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -191> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -190> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -189> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -188> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -187> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -186> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -185> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: crc 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -184> 2023-12-03T11:37:44.613+0000 7f7e50d42800 5 AuthRegistry(0x7ffc2dcac920) adding con mode: secure 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -183> 2023-12-03T11:37:44.614+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -182> 2023-12-03T11:37:44.614+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -181> 2023-12-03T11:37:44.614+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x7ffc2dcaca68 EEXIST 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -180> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.728 INFO:tasks.workunit.client.0.smithi012.stderr: -179> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -178> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f434a07d0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -177> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f4349f7e0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -176> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f4349b4b0 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -175> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -174> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -173> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -172> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -171> 2023-12-03T11:37:44.614+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.796984s 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -170> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f434a07d0 auth_method 0 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -169> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -168> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -167> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -166> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f4349f7e0 auth_method 0 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -165> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -164> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.729 INFO:tasks.workunit.client.0.smithi012.stderr: -163> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -162> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f4349b4b0 auth_method 0 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -161> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -160> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -159> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -158> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -157> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -156> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -155> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -154> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -153> 2023-12-03T11:37:44.614+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -152> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -151> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -150> 2023-12-03T11:37:44.614+0000 7f7e4c302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -149> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_done global_id 7254 payload 274 2023-12-03T11:37:44.730 INFO:tasks.workunit.client.0.smithi012.stderr: -148> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -147> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -146> 2023-12-03T11:37:44.614+0000 7f7e4cb03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -145> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -144> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -143> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: dump: 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.731 INFO:tasks.workunit.client.0.smithi012.stderr: -142> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -141> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -140> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -139> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7254 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -138> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -137> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: -136> 2023-12-03T11:37:44.615+0000 7f7e127e4640 10 monclient: dump: 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.732 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -135> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: get_monmap_and_config success 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -134> 2023-12-03T11:37:44.615+0000 7f7e50d42800 4 set_mon_vals no callback set 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -133> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: shutdown 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -132> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: build_initial_monmap 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -131> 2023-12-03T11:37:44.615+0000 7f7e50d42800 1 build_initial for_mkfs: 0 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr: -130> 2023-12-03T11:37:44.615+0000 7f7e50d42800 10 monclient: monmap: 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 0 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 0.000000 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:created 0.000000 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 0 (unknown) 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.noname-a 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.noname-b 2023-12-03T11:37:44.733 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.noname-c 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -129> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -128> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -127> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding auth protocol: cephx 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -126> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -125> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -124> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -123> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -122> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -121> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -120> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -119> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -118> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -117> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.734 INFO:tasks.workunit.client.0.smithi012.stderr: -116> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: crc 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -115> 2023-12-03T11:37:44.615+0000 7f7e50d42800 5 AuthRegistry(0x562f4349de50) adding con mode: secure 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -114> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -113> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: starting msgr at 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -112> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: starting objecter 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -111> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command objecter_requests cmddesc objecter_requests hook 0x562f43460430 EEXIST 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -110> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: setting wanted keys 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -109> 2023-12-03T11:37:44.616+0000 7f7e50d42800 1 librados: calling monclient init 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -108> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: init 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -107> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -106> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -105> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding auth protocol: cephx 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -104> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -103> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -102> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.735 INFO:tasks.workunit.client.0.smithi012.stderr: -101> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -100> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -99> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -98> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -97> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -96> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -95> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -94> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: crc 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -93> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 AuthRegistry(0x562f434a1478) adding con mode: secure 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -92> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -91> 2023-12-03T11:37:44.616+0000 7f7e50d42800 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.keyring 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -90> 2023-12-03T11:37:44.616+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command rotate-key cmddesc rotate-key hook 0x562f434a15c0 EEXIST 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -89> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: _reopen_session rank -1 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -88> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient: _add_conns ranks=[2,0,1] 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -87> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-c con 0x562f4349b4b0 addr [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] 2023-12-03T11:37:44.736 INFO:tasks.workunit.client.0.smithi012.stderr: -86> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-a con 0x562f434a07d0 addr [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -85> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): picked mon.noname-b con 0x562f434b3700 addr [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -84> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -83> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -82> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): start opening mon connection 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -81> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): _renew_subs 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -80> 2023-12-03T11:37:44.616+0000 7f7e50d42800 10 monclient(hunting): authenticate will time out at 1674.799485s 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -79> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): get_auth_request con 0x562f4349b4b0 auth_method 0 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -78> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -77> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -76> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -75> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request con 0x562f434a07d0 auth_method 0 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -74> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -73> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -72> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.737 INFO:tasks.workunit.client.0.smithi012.stderr: -71> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -70> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -69> 2023-12-03T11:37:44.617+0000 7f7e4d304640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -68> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -67> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -66> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -65> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): get_auth_request con 0x562f434b3700 auth_method 0 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -64> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -63> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): _init_auth method 2 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -62> 2023-12-03T11:37:44.617+0000 7f7e4c302640 10 monclient(hunting): _init_auth creating new auth 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -61> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient(hunting): handle_auth_done global_id 7257 payload 824 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -60> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient: _finish_hunting 0 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -59> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 1 monclient: found mon.noname-a 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -58> 2023-12-03T11:37:44.617+0000 7f7e4cb03640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -57> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.738 INFO:tasks.workunit.client.0.smithi012.stderr: -56> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: -55> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: dump: 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: -54> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: _finish_auth 0 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: -53> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: _check_auth_tickets 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: -52> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_config config(0 keys) v1 2023-12-03T11:37:44.739 INFO:tasks.workunit.client.0.smithi012.stderr: -51> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -50> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -49> 2023-12-03T11:37:44.617+0000 7f7e10fe1640 10 monclient: dump: 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:epoch 1 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:fsid a018b7e2-c099-40b1-a133-6e1bf20b2274 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:last_changed 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:created 2023-12-03T11:23:18.718260+0000 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:min_mon_release 18 (reef) 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:election_strategy: 1 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:1: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.b 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr:2: [v2:172.21.15.12:3301/0,v1:172.21.15.12:6790/0] mon.c 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -48> 2023-12-03T11:37:44.617+0000 7f7e50d42800 5 monclient: authenticate success, global_id 7257 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -47> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -46> 2023-12-03T11:37:44.617+0000 7f7e127e4640 4 set_mon_vals no callback set 2023-12-03T11:37:44.740 INFO:tasks.workunit.client.0.smithi012.stderr: -45> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -44> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _renew_subs 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -43> 2023-12-03T11:37:44.617+0000 7f7e50d42800 10 monclient: _send_mon_message to mon.a at v2:172.21.15.12:3300/0 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -42> 2023-12-03T11:37:44.617+0000 7f7e50d42800 1 librados: init done 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -41> 2023-12-03T11:37:44.617+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command cr dump hook 0x562f434a6c18 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -40> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc handle_mgr_map Got map version 6 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -39> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc handle_mgr_map Active mgr is now [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -38> 2023-12-03T11:37:44.618+0000 7f7e10fe1640 4 mgrc reconnect Starting new session with [v2:172.21.15.12:6832/3565462874,v1:172.21.15.12:6833/3565462874] 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -37> 2023-12-03T11:37:44.619+0000 7f7e4d304640 10 monclient: get_auth_request con 0x7f7d88069820 auth_method 0 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -36> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 note: GC not initialized 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -35> 2023-12-03T11:37:44.619+0000 7f7d81ffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -34> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace show hook 0x7f7e40004460 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -33> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace history hook 0x7f7e40004460 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -32> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace active hook 0x7f7e40004460 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -31> 2023-12-03T11:37:44.619+0000 7f7e50d42800 5 asok(0x562f43202a10) register_command sync trace active_short hook 0x7f7e40004460 2023-12-03T11:37:44.741 INFO:tasks.workunit.client.0.smithi012.stderr: -30> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -29> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 Filter name: none 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -28> 2023-12-03T11:37:44.619+0000 7f7d737fe640 20 reqs_thread_entry: start 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -27> 2023-12-03T11:37:44.619+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -26> 2023-12-03T11:37:44.619+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -25> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -24> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -23> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -22> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 : miss 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -21> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -20> 2023-12-03T11:37:44.620+0000 7f7e50d42800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -19> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -18> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -17> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -16> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.742 INFO:tasks.workunit.client.0.smithi012.stderr: -15> 2023-12-03T11:37:44.620+0000 7f7e50d42800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -14> 2023-12-03T11:37:44.621+0000 7f7e4c302640 10 monclient: get_auth_request con 0x562f434ef3f0 auth_method 0 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -13> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -12> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -11> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -10> 2023-12-03T11:37:44.625+0000 7f7e50d42800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27, ofs0, ns] 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -9> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725963 cookie 94760992107232 notifier 7251 bl.length()=873 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -8> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -7> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -6> 2023-12-03T11:37:44.625+0000 7f7e337fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -5> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_sources= new_sources= 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -4> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential sources added=[] removed=[] 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -3> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_dests= new_dests= 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -2> 2023-12-03T11:37:44.626+0000 7f7e50d42800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential dests added=[] removed=[] 2023-12-03T11:37:44.743 INFO:tasks.workunit.client.0.smithi012.stderr: -1> 2023-12-03T11:37:44.627+0000 7f7e50d42800 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: In function 'int FaultInjector::check(const Key&) const::visitor::operator()(const InjectAbort&) const [with Key = std::basic_string_view]' thread 7f7e50d42800 time 2023-12-03T11:37:44.628187+0000 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7537-gbcbf278c/rpm/el9/BUILD/ceph-18.0.0-7537-gbcbf278c/src/common/fault_injector.h: 120: FAILED ceph_assert(!"FaultInjector") 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7f7e5396eef8] 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 3: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 4: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 5: main() 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 7: __libc_start_main() 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 8: _start() 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 0> 2023-12-03T11:37:44.628+0000 7f7e50d42800 -1 *** Caught signal (Aborted) ** 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: in thread 7f7e50d42800 thread_name:radosgw-admin 2023-12-03T11:37:44.744 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7f7e51254db0] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7f7e512a154c] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 3: raise() 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 4: abort() 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f7e5396ef52] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16f0b6) [0x7f7e5396f0b6] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x562f417b2a71] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 8: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, optional_yield, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x314) [0x562f417bef34] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 9: main() 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 10: /lib64/libc.so.6(+0x3feb0) [0x7f7e5123feb0] 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 11: __libc_start_main() 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 12: _start() 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.745 INFO:tasks.workunit.client.0.smithi012.stderr:--- logging levels --- 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 none 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 lockdep 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 context 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 crush 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_balancer 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_locker 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_log_expire 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mds_migrator 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 buffer 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 timer 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 filer 2023-12-03T11:37:44.746 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 striper 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 1 objecter 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rados 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_mirror 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_replay 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 rbd_pwl 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 journaler 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objectcacher 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 immutable_obj_cache 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 client 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 osd 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 optracker 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 objclass 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 filestore 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 journal 2023-12-03T11:37:44.747 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 ms 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mon 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 0/10 monc 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 paxos 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 tp 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 auth 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 crypto 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 finisher 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 reserver 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 heartbeatmap 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 perfcounter 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 20/20 rgw 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_sync 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_datacache 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_access 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_dbstore 2023-12-03T11:37:44.748 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 rgw_flight 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 javaclient 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 asok 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 1 throttle 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 0 refs 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 compressor 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluestore 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 bluefs 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 3 bdev 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 kstore 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 4/ 5 rocksdb 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 fuse 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 2/ 5 mgr 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mgrc 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 dpdk 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 eventtrace 2023-12-03T11:37:44.749 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 prioritycache 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 test 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephfs_mirror 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cephsqlite 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 crimson_interrupt 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_onode 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_odata 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_omap 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_tm 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_t 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cleaner 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_epm 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_lba 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_fixedkv_tree 2023-12-03T11:37:44.750 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_cache 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_journal 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_device 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 seastore_backref 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 alienstore 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 mclock 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 0/ 5 cyanstore 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 ceph_exporter 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 memstore 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 1/ 5 trace 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: -2/-2 (syslog threshold) 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 99/99 (stderr threshold) 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr:--- pthread ID / name mapping for recent threads --- 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7d737fe640 / http_manager 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7d81ffb640 / http_manager 2023-12-03T11:37:44.751 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e10fe1640 / ms_dispatch 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e127e4640 / io_context_pool 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e12fe5640 / rgw_dt_lg_renew 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e31ffb640 / ms_dispatch 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e337fe640 / io_context_pool 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4a2fe640 / ms_dispatch 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4b300640 / io_context_pool 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4c302640 / msgr-worker-2 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4cb03640 / msgr-worker-1 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e4d304640 / msgr-worker-0 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: 7f7e50d42800 / radosgw-admin 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: max_recent 500 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: max_new 1000 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr: log_file /var/lib/ceph/crash/2023-12-03T11:37:44.629595Z_36bfaeae-3694-4419-ab08-efdba5f9d7da/log 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr:--- end dump of recent events --- 2023-12-03T11:37:44.752 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:37:44.753 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:37:44.753 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:37:44.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.679+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 20 realm 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.881+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.882+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.882+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.883+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.883+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.884+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:37:44.885 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.884+0000 7f201b4c0800 20 searching for the correct realm 2023-12-03T11:37:44.893 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.892+0000 7f20197c1640 0 -- 172.21.15.12:0/1571707549 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5614135b6d10 msgr2=0x5614135b91d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.894+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 zone default found 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 4 Realm: () 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 zonegroup default 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.895+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.899 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.896+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.896+0000 7f201b4c0800 20 started sync module instance, tier type = 2023-12-03T11:37:44.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.896+0000 7f201b4c0800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:37:44.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.900+0000 7f20187bf640 0 -- 172.21.15.12:0/1571707549 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x5614135b2450 msgr2=0x5614135b4840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=0 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=1 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=2 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=3 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=4 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=5 2023-12-03T11:37:44.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.905+0000 7f201b4c0800 20 add_watcher() i=6 2023-12-03T11:37:44.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.906+0000 7f201b4c0800 20 add_watcher() i=7 2023-12-03T11:37:44.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.906+0000 7f201b4c0800 2 all 8 watchers are set, enabling cache 2023-12-03T11:37:44.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.908+0000 7f1fdefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.908+0000 7f201b4c0800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:37:44.909 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.908+0000 7f201b4c0800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:37:44.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.914+0000 7f201b4c0800 5 note: GC not initialized 2023-12-03T11:37:44.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.914+0000 7f1f4dffb640 20 reqs_thread_entry: start 2023-12-03T11:37:44.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.914+0000 7f201b4c0800 20 init_complete bucket index max shards: 1 2023-12-03T11:37:44.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.914+0000 7f201b4c0800 20 Filter name: none 2023-12-03T11:37:44.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.914+0000 7f1f3f7fe640 20 reqs_thread_entry: start 2023-12-03T11:37:44.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.915+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.915+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.916 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.915+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.915+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=403 2023-12-03T11:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.915+0000 7f201b4c0800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.916+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.916+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:37:44.917 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.916+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.916+0000 7f201b4c0800 20 rados_obj.operate() r=0 bl.length=403 2023-12-03T11:37:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.916+0000 7f201b4c0800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:37:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.917+0000 7f201b4c0800 20 rados->read ofs=0 len=0 2023-12-03T11:37:44.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.917+0000 7f201b4c0800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:37:44.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.918+0000 7f1fdefe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:37:44.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.920+0000 7f201b4c0800 20 remove_watcher() i=0 2023-12-03T11:37:44.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.920+0000 7f201b4c0800 2 removed watcher, disabling cache 2023-12-03T11:37:44.922 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.921+0000 7f201b4c0800 20 remove_watcher() i=1 2023-12-03T11:37:44.923 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.922+0000 7f201b4c0800 20 remove_watcher() i=2 2023-12-03T11:37:44.924 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.923+0000 7f201b4c0800 20 remove_watcher() i=3 2023-12-03T11:37:44.925 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.924+0000 7f201b4c0800 20 remove_watcher() i=4 2023-12-03T11:37:44.926 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.925+0000 7f201b4c0800 20 remove_watcher() i=5 2023-12-03T11:37:44.927 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.926+0000 7f201b4c0800 20 remove_watcher() i=6 2023-12-03T11:37:44.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:37:44.927+0000 7f201b4c0800 20 remove_watcher() i=7 2023-12-03T11:37:44.933 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:37:44.933 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:37:44.933 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 num_objects: 20 size_kb: 1 num_shards: 1 2023-12-03T11:37:44.933 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.016 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket abort-at-do-reshard --num-shards 2 --rgw-reshard-bucket-lock-duration 30 2023-12-03T11:38:15.051 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.051 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:15.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.067+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.068+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.068+0000 7ff1e7b28800 20 realm 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.068+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.069+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.069+0000 7ff1e7b28800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.069+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.069+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.069+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.070+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.070+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.071+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:15.072 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.071+0000 7ff1e7b28800 20 searching for the correct realm 2023-12-03T11:38:15.083 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.082+0000 7ff1e37d0640 0 -- 172.21.15.12:0/2293353984 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7ff1d801a040 msgr2=0x7ff1d8026990 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:15.085 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.084+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.086 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 zone default found 2023-12-03T11:38:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 4 Realm: () 2023-12-03T11:38:15.087 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 zonegroup default 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.085+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.086+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.086+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.086+0000 7ff1e7b28800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.086+0000 7ff1e7b28800 20 started sync module instance, tier type = 2023-12-03T11:38:15.088 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.086+0000 7ff1e7b28800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:15.093 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.092+0000 7ff1e37d0640 0 -- 172.21.15.12:0/2293353984 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7ff1bc079d80 msgr2=0x7ff1bc07c1b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=0 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=1 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=2 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=3 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=4 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=5 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.096+0000 7ff1e7b28800 20 add_watcher() i=6 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.097+0000 7ff1e7b28800 20 add_watcher() i=7 2023-12-03T11:38:15.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.097+0000 7ff1e7b28800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:15.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.099+0000 7ff1a97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.100+0000 7ff1e7b28800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:15.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.100+0000 7ff1e7b28800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.106+0000 7ff1e7b28800 5 note: GC not initialized 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.106+0000 7ff117fff640 20 reqs_thread_entry: start 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 20 Filter name: none 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff115ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 10 cache get: name=default.rgw.meta+root+abort-at-do-reshard : miss 2023-12-03T11:38:15.108 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 10 cache put: name=default.rgw.meta+root+abort-at-do-reshard info.flags=0x11 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 10 adding default.rgw.meta+root+abort-at-do-reshard to cache LRU end 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 : miss 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.107+0000 7ff1e7b28800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.109 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:38:15.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:38:15.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 10 adding default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:38:15.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:15.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:38:15.110 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.108+0000 7ff1e7b28800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27, ofs0, ns] 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1c9ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725965 cookie 94622806115024 notifier 6920 bl.length()=873 2023-12-03T11:38:15.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1c9ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1c9ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1c9ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_sources= new_sources= 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential sources added=[] removed=[] 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_dests= new_dests= 2023-12-03T11:38:15.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.113+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential dests added=[] removed=[] 2023-12-03T11:38:15.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.115+0000 7ff1e7b28800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27.1.0 2023-12-03T11:38:15.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.115+0000 7ff1e7b28800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27.1.0 2023-12-03T11:38:15.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.115+0000 7ff1e7b28800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27.1.1 2023-12-03T11:38:15.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.115+0000 7ff1e7b28800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27.1.1 2023-12-03T11:38:15.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.115+0000 7ff1e7b28800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27.0 2023-12-03T11:38:15.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.117+0000 7ff1e7b28800 10 cache get: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 : hit (requested=0x17, cached=0x17) 2023-12-03T11:38:15.118 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.117+0000 7ff1e7b28800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 10 cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 10 moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 10 updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 10 distributing notification oid=default.rgw.control:notify.1 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27, ofs0, ns] 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1c97fa640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725966 cookie 94622806115024 notifier 6920 bl.length()=841 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1c97fa640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 info.flags=0x17 2023-12-03T11:38:15.120 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1c97fa640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.abort-at-do-reshard:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 to cache LRU end 2023-12-03T11:38:15.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1c97fa640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=205 2023-12-03T11:38:15.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_sources= new_sources= 2023-12-03T11:38:15.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential sources added=[] removed=[] 2023-12-03T11:38:15.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): orig_dests= new_dests= 2023-12-03T11:38:15.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.119+0000 7ff1e7b28800 20 handle_bi_update(): bucket=:abort-at-do-reshard[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27]): potential dests added=[] removed=[] 2023-12-03T11:38:15.122 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.121+0000 7ff1e7b28800 1 execute INFO: reshard of bucket "abort-at-do-reshard" completed successfully 2023-12-03T11:38:15.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.123+0000 7ff1a97e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.126 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.125+0000 7ff1e7b28800 20 remove_watcher() i=0 2023-12-03T11:38:15.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.125+0000 7ff1e7b28800 2 removed watcher, disabling cache 2023-12-03T11:38:15.128 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.127+0000 7ff1e7b28800 20 remove_watcher() i=1 2023-12-03T11:38:15.129 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.128+0000 7ff1e7b28800 20 remove_watcher() i=2 2023-12-03T11:38:15.130 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.129+0000 7ff1e7b28800 20 remove_watcher() i=3 2023-12-03T11:38:15.131 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.130+0000 7ff1e7b28800 20 remove_watcher() i=4 2023-12-03T11:38:15.132 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.131+0000 7ff1e7b28800 20 remove_watcher() i=5 2023-12-03T11:38:15.133 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.132+0000 7ff1e7b28800 20 remove_watcher() i=6 2023-12-03T11:38:15.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.133+0000 7ff1e7b28800 20 remove_watcher() i=7 2023-12-03T11:38:15.141 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket abort-at-do-reshard 2023-12-03T11:38:15.169 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.169 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:15.184 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.183+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.186 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 realm 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.185+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.187 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.186+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.186+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.187+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:15.188 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.187+0000 7f4f07f26800 20 searching for the correct realm 2023-12-03T11:38:15.197 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.196+0000 7f4f04324640 0 -- 172.21.15.12:0/1059320216 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5642135ff800 msgr2=0x564213601cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:15.199 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.200 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.198+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 zone default found 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 4 Realm: () 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 zonegroup default 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.201 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.199+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.200+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.200+0000 7f4f07f26800 20 started sync module instance, tier type = 2023-12-03T11:38:15.202 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.200+0000 7f4f07f26800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:15.205 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.204+0000 7f4f03b23640 0 -- 172.21.15.12:0/1059320216 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7f4eec03b040 msgr2=0x7f4eec023430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=0 2023-12-03T11:38:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=1 2023-12-03T11:38:15.209 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=2 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=3 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=4 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=5 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.208+0000 7f4f07f26800 20 add_watcher() i=6 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.209+0000 7f4f07f26800 20 add_watcher() i=7 2023-12-03T11:38:15.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.209+0000 7f4f07f26800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:15.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.211+0000 7f4ec9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.211+0000 7f4f07f26800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:15.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.211+0000 7f4f07f26800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.217+0000 7f4f07f26800 5 note: GC not initialized 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.217+0000 7f4e40ff9640 20 reqs_thread_entry: start 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4f07f26800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4f07f26800 20 Filter name: none 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4e367fc640 20 reqs_thread_entry: start 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.219 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:38:15.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.218+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.219+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:38:15.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.219+0000 7f4f07f26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.220 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.219+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.219+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=187 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.219+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.220+0000 7f4f07f26800 20 rados_obj.operate() r=0 bl.length=371 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.220+0000 7f4f07f26800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.220+0000 7f4f07f26800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.221 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.220+0000 7f4f07f26800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.222 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.221+0000 7f4ec9fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.224+0000 7f4f07f26800 20 remove_watcher() i=0 2023-12-03T11:38:15.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.224+0000 7f4f07f26800 2 removed watcher, disabling cache 2023-12-03T11:38:15.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.225+0000 7f4f07f26800 20 remove_watcher() i=1 2023-12-03T11:38:15.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.226+0000 7f4f07f26800 20 remove_watcher() i=2 2023-12-03T11:38:15.228 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.227+0000 7f4f07f26800 20 remove_watcher() i=3 2023-12-03T11:38:15.229 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.228+0000 7f4f07f26800 20 remove_watcher() i=4 2023-12-03T11:38:15.230 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.229+0000 7f4f07f26800 20 remove_watcher() i=5 2023-12-03T11:38:15.231 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.230+0000 7f4f07f26800 20 remove_watcher() i=6 2023-12-03T11:38:15.232 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.231+0000 7f4f07f26800 20 remove_watcher() i=7 2023-12-03T11:38:15.236 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.237 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:38:15.237 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: abort-at-do-reshard id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27 num_objects: 19 size_kb: 1 num_shards: 2 2023-12-03T11:38:15.237 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.321 INFO:tasks.workunit.client.0.smithi012.stderr: test: reshard versioned bucket 2023-12-03T11:38:15.323 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:38:15.351 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.351 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:15.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.365+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 realm 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.367+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.369 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.368+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.368+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.369+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:15.370 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.369+0000 7fe16cb23800 20 searching for the correct realm 2023-12-03T11:38:15.380 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.379+0000 7fe1688e5640 0 -- 172.21.15.12:0/3563401683 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x563779249440 msgr2=0x56377924b830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:15.382 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 zone default found 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 4 Realm: () 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 zonegroup default 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.380+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.383 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 started sync module instance, tier type = 2023-12-03T11:38:15.384 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.381+0000 7fe16cb23800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:15.389 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.388+0000 7fe1690e6640 0 -- 172.21.15.12:0/3563401683 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fe15c069430 msgr2=0x7fe15c069890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=0 2023-12-03T11:38:15.391 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=1 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=2 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=3 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=4 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=5 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.390+0000 7fe16cb23800 20 add_watcher() i=6 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.391+0000 7fe16cb23800 20 add_watcher() i=7 2023-12-03T11:38:15.392 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.391+0000 7fe16cb23800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:15.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.393+0000 7fe12efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.394 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.393+0000 7fe16cb23800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:15.395 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.393+0000 7fe16cb23800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.399+0000 7fe16cb23800 5 note: GC not initialized 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe09dffb640 20 reqs_thread_entry: start 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe16cb23800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe16cb23800 20 Filter name: none 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe08f7fe640 20 reqs_thread_entry: start 2023-12-03T11:38:15.401 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:15.402 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.400+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.401+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.401+0000 7fe16cb23800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.401+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.401+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.401+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.403 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.402+0000 7fe16cb23800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:15.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.402+0000 7fe16cb23800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.402+0000 7fe16cb23800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.402+0000 7fe16cb23800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.404 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.403+0000 7fe12efe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.405+0000 7fe16cb23800 20 remove_watcher() i=0 2023-12-03T11:38:15.406 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.405+0000 7fe16cb23800 2 removed watcher, disabling cache 2023-12-03T11:38:15.407 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.406+0000 7fe16cb23800 20 remove_watcher() i=1 2023-12-03T11:38:15.408 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.407+0000 7fe16cb23800 20 remove_watcher() i=2 2023-12-03T11:38:15.410 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.409+0000 7fe16cb23800 20 remove_watcher() i=3 2023-12-03T11:38:15.411 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.410+0000 7fe16cb23800 20 remove_watcher() i=4 2023-12-03T11:38:15.412 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.411+0000 7fe16cb23800 20 remove_watcher() i=5 2023-12-03T11:38:15.413 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.412+0000 7fe16cb23800 20 remove_watcher() i=6 2023-12-03T11:38:15.414 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.413+0000 7fe16cb23800 20 remove_watcher() i=7 2023-12-03T11:38:15.418 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.419 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:38:15.419 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: myver id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 num_objects: 0 size_kb: 0 num_shards: 1 2023-12-03T11:38:15.419 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.420 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 2 2023-12-03T11:38:15.450 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.450 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:15.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.463+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 20 realm 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.465+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.466+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.466+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.467+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.467+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.467+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:15.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.467+0000 7f6f53531800 20 searching for the correct realm 2023-12-03T11:38:15.478 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.477+0000 7f6f4f1d7640 0 -- 172.21.15.12:0/2482916267 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55ca330bcdf0 msgr2=0x55ca330bf220 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:15.479 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:15.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:15.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:15.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.478+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 zone default found 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 4 Realm: () 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:15.481 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 zonegroup default 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.479+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.480+0000 7f6f53531800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.480+0000 7f6f53531800 20 started sync module instance, tier type = 2023-12-03T11:38:15.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.480+0000 7f6f53531800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:15.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.486+0000 7f6f4f9d8640 0 -- 172.21.15.12:0/2482916267 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f6f4405e040 msgr2=0x7f6f440750b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=0 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=1 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=2 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=3 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=4 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=5 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.488+0000 7f6f53531800 20 add_watcher() i=6 2023-12-03T11:38:15.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.489+0000 7f6f53531800 20 add_watcher() i=7 2023-12-03T11:38:15.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.489+0000 7f6f53531800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:15.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.491+0000 7f6f157e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.491+0000 7f6f53531800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:15.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.491+0000 7f6f53531800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.497+0000 7f6f53531800 5 note: GC not initialized 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.497+0000 7f6e83fff640 20 reqs_thread_entry: start 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.497+0000 7f6f53531800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.497+0000 7f6f53531800 20 Filter name: none 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.497+0000 7f6e81ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:38:15.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : miss 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 adding default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:15.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.498+0000 7f6f53531800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:15.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:15.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:15.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:15.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f35ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725945 cookie 94326928226496 notifier 6983 bl.length()=783 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f35ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f35ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f35ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:15.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.503+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.0 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.0 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.1 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.1 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.0 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:38:15.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.505+0000 7f6f53531800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f53531800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f53531800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f53531800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f53531800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f35ffb640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725946 cookie 94326928226496 notifier 6983 bl.length()=751 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f35ffb640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f35ffb640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.506+0000 7f6f35ffb640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:15.508 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.507+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:15.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.507+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:15.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.507+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:15.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.507+0000 7f6f53531800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:15.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.508+0000 7f6f53531800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:38:15.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.510+0000 7f6f157e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.512+0000 7f6f53531800 20 remove_watcher() i=0 2023-12-03T11:38:15.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.512+0000 7f6f53531800 2 removed watcher, disabling cache 2023-12-03T11:38:15.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.513+0000 7f6f53531800 20 remove_watcher() i=1 2023-12-03T11:38:15.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.514+0000 7f6f53531800 20 remove_watcher() i=2 2023-12-03T11:38:15.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.515+0000 7f6f53531800 20 remove_watcher() i=3 2023-12-03T11:38:15.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.516+0000 7f6f53531800 20 remove_watcher() i=4 2023-12-03T11:38:15.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.517+0000 7f6f53531800 20 remove_watcher() i=5 2023-12-03T11:38:15.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.518+0000 7f6f53531800 20 remove_watcher() i=6 2023-12-03T11:38:15.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.519+0000 7f6f53531800 20 remove_watcher() i=7 2023-12-03T11:38:15.526 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:38:15.553 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.554 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:15.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.566+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 realm 2023-12-03T11:38:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.569 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.568+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.569+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.570 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.569+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.570+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:15.571 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.570+0000 7f7b39412800 20 searching for the correct realm 2023-12-03T11:38:15.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b34b2a640 0 -- 172.21.15.12:0/2719174836 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55eff699cc00 msgr2=0x55eff699eff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b39412800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b39412800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b39412800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b39412800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.580+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.581+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.581+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:15.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:15.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 zone default found 2023-12-03T11:38:15.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 4 Realm: () 2023-12-03T11:38:15.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:15.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:15.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:15.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 zonegroup default 2023-12-03T11:38:15.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.782+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.783+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.783+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.783+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.783+0000 7f7b39412800 20 started sync module instance, tier type = 2023-12-03T11:38:15.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.783+0000 7f7b39412800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:15.791 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.790+0000 7f7b3532b640 0 -- 172.21.15.12:0/2719174836 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f7b28038970 msgr2=0x7f7b28035430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=0 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=1 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=2 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=3 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=4 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=5 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.792+0000 7f7b39412800 20 add_watcher() i=6 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.793+0000 7f7b39412800 20 add_watcher() i=7 2023-12-03T11:38:15.794 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.793+0000 7f7b39412800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:15.796 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.795+0000 7f7afafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.795+0000 7f7b39412800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:15.797 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.795+0000 7f7b39412800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:15.804 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.802+0000 7f7b39412800 5 note: GC not initialized 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.802+0000 7f7a69ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.802+0000 7f7b39412800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.803+0000 7f7b39412800 20 Filter name: none 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.803+0000 7f7a5b7fe640 20 reqs_thread_entry: start 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.803+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.803+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:15.805 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.803+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.806 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.804+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.804+0000 7f7b39412800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.804+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.804+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.804+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.805+0000 7f7b39412800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.805+0000 7f7b39412800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.805+0000 7f7b39412800 20 rados->read ofs=0 len=0 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.805+0000 7f7b39412800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:15.807 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.806+0000 7f7afafe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:15.809 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.808+0000 7f7b39412800 20 remove_watcher() i=0 2023-12-03T11:38:15.810 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.808+0000 7f7b39412800 2 removed watcher, disabling cache 2023-12-03T11:38:15.811 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.810+0000 7f7b39412800 20 remove_watcher() i=1 2023-12-03T11:38:15.812 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.811+0000 7f7b39412800 20 remove_watcher() i=2 2023-12-03T11:38:15.813 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.812+0000 7f7b39412800 20 remove_watcher() i=3 2023-12-03T11:38:15.814 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.813+0000 7f7b39412800 20 remove_watcher() i=4 2023-12-03T11:38:15.815 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.814+0000 7f7b39412800 20 remove_watcher() i=5 2023-12-03T11:38:15.816 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.815+0000 7f7b39412800 20 remove_watcher() i=6 2023-12-03T11:38:15.817 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:15.816+0000 7f7b39412800 20 remove_watcher() i=7 2023-12-03T11:38:15.822 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.822 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:38:15.822 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: myver id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 num_objects: 0 size_kb: 0 num_shards: 2 2023-12-03T11:38:15.822 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:15.868 INFO:tasks.workunit.client.0.smithi012.stderr: test: reshard removes olh entries with empty name 2023-12-03T11:38:15.959 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket a-bucket 2023-12-03T11:38:15.987 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:15.987 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.004 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.003+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 realm 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.006 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.005+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.006+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.007 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.006+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.007+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.008 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.007+0000 7fd2e9b3a800 20 searching for the correct realm 2023-12-03T11:38:16.017 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.016+0000 7fd2e5806640 0 -- 172.21.15.12:0/15118318 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x564cfe61ce20 msgr2=0x564cfe61f210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.017+0000 7fd2e9b3a800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.017+0000 7fd2e9b3a800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.017+0000 7fd2e9b3a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.017+0000 7fd2e9b3a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.017+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.019 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 zone default found 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 4 Realm: () 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 zonegroup default 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.018+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.019+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.019+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.019+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.021 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.019+0000 7fd2e9b3a800 20 started sync module instance, tier type = 2023-12-03T11:38:16.022 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.019+0000 7fd2e9b3a800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.027 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.025+0000 7fd2e6007640 0 -- 172.21.15.12:0/15118318 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x564cfe624e90 msgr2=0x564cfe627320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=0 2023-12-03T11:38:16.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=1 2023-12-03T11:38:16.029 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=2 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=3 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=4 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=5 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.028+0000 7fd2e9b3a800 20 add_watcher() i=6 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.029+0000 7fd2e9b3a800 20 add_watcher() i=7 2023-12-03T11:38:16.030 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.029+0000 7fd2e9b3a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.031+0000 7fd2ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.031+0000 7fd2e9b3a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.032 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.031+0000 7fd2e9b3a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.038 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd2e9b3a800 5 note: GC not initialized 2023-12-03T11:38:16.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd21a7fc640 20 reqs_thread_entry: start 2023-12-03T11:38:16.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd2e9b3a800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd2e9b3a800 20 Filter name: none 2023-12-03T11:38:16.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd20bfff640 20 reqs_thread_entry: start 2023-12-03T11:38:16.039 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.037+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.038+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.040 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.039+0000 7fd2e9b3a800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.039+0000 7fd2e9b3a800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.039+0000 7fd2e9b3a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.040+0000 7fd2e9b3a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.041 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.040+0000 7fd2ab7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.042+0000 7fd2e9b3a800 20 remove_watcher() i=0 2023-12-03T11:38:16.043 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.042+0000 7fd2e9b3a800 2 removed watcher, disabling cache 2023-12-03T11:38:16.044 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.043+0000 7fd2e9b3a800 20 remove_watcher() i=1 2023-12-03T11:38:16.045 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.044+0000 7fd2e9b3a800 20 remove_watcher() i=2 2023-12-03T11:38:16.046 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.045+0000 7fd2e9b3a800 20 remove_watcher() i=3 2023-12-03T11:38:16.047 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.046+0000 7fd2e9b3a800 20 remove_watcher() i=4 2023-12-03T11:38:16.048 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.047+0000 7fd2e9b3a800 20 remove_watcher() i=5 2023-12-03T11:38:16.049 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.048+0000 7fd2e9b3a800 20 remove_watcher() i=6 2023-12-03T11:38:16.050 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.049+0000 7fd2e9b3a800 20 remove_watcher() i=7 2023-12-03T11:38:16.055 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:16.055 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:38:16.055 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: a-bucket id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 num_objects: 0 size_kb: 0 num_shards: 3 2023-12-03T11:38:16.055 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:16.056 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket layout --bucket a-bucket 2023-12-03T11:38:16.085 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.086 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.097+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 realm 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.099+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.100+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.100+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.101+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.101+0000 7f230fd3d800 20 searching for the correct realm 2023-12-03T11:38:16.111 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.110+0000 7f230baa3640 0 -- 172.21.15.12:0/3398921178 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x564a5445f150 msgr2=0x564a54461560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:38:16.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.113 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.112+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.114 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 zone default found 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 4 Realm: () 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 zonegroup default 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.115 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.113+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.114+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.114+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.114+0000 7f230fd3d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.114+0000 7f230fd3d800 20 started sync module instance, tier type = 2023-12-03T11:38:16.116 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.114+0000 7f230fd3d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.121 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.120+0000 7f230c2a4640 0 -- 172.21.15.12:0/3398921178 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x564a54467250 msgr2=0x564a54469710 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=0 2023-12-03T11:38:16.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=1 2023-12-03T11:38:16.124 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=2 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=3 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=4 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=5 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.123+0000 7f230fd3d800 20 add_watcher() i=6 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.124+0000 7f230fd3d800 20 add_watcher() i=7 2023-12-03T11:38:16.125 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.124+0000 7f230fd3d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.126+0000 7f22d1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.126+0000 7f230fd3d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.127 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.126+0000 7f230fd3d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.132+0000 7f230fd3d800 5 note: GC not initialized 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.132+0000 7f2248ff9640 20 reqs_thread_entry: start 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.132+0000 7f230fd3d800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.132+0000 7f230fd3d800 20 Filter name: none 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.133+0000 7f223e7fc640 20 reqs_thread_entry: start 2023-12-03T11:38:16.134 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.133+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.133+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.133+0000 7f230fd3d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.134+0000 7f230fd3d800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.134+0000 7f230fd3d800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.135 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.134+0000 7f22d1fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.136+0000 7f230fd3d800 20 remove_watcher() i=0 2023-12-03T11:38:16.137 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.136+0000 7f230fd3d800 2 removed watcher, disabling cache 2023-12-03T11:38:16.138 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.137+0000 7f230fd3d800 20 remove_watcher() i=1 2023-12-03T11:38:16.139 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.138+0000 7f230fd3d800 20 remove_watcher() i=2 2023-12-03T11:38:16.140 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.139+0000 7f230fd3d800 20 remove_watcher() i=3 2023-12-03T11:38:16.141 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.140+0000 7f230fd3d800 20 remove_watcher() i=4 2023-12-03T11:38:16.142 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.141+0000 7f230fd3d800 20 remove_watcher() i=5 2023-12-03T11:38:16.143 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.142+0000 7f230fd3d800 20 remove_watcher() i=6 2023-12-03T11:38:16.144 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.143+0000 7f230fd3d800 20 remove_watcher() i=7 2023-12-03T11:38:16.151 INFO:tasks.workunit.client.0.smithi012.stderr:rados -p default.rgw.buckets.index setomapval .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 --omap-key-file /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_key < /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/olh_noname_val 2023-12-03T11:38:16.176 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.176 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.196 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket a-bucket 2023-12-03T11:38:16.225 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.226 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.240 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.239+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.240+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.240+0000 7fe803922800 20 realm 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.240+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.241+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.241+0000 7fe803922800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.242 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.241+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.241+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.241+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.242+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.243 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.242+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.242+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.244 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.242+0000 7fe803922800 20 searching for the correct realm 2023-12-03T11:38:16.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.251+0000 7fe7ffdca640 0 -- 172.21.15.12:0/4209094071 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55b68d5291a0 msgr2=0x55b68d52b630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.253+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 zone default found 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 4 Realm: () 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 zonegroup default 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.254+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.255+0000 7fe803922800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.255+0000 7fe803922800 20 started sync module instance, tier type = 2023-12-03T11:38:16.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.255+0000 7fe803922800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.260 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.259+0000 7fe7ff5c9640 0 -- 172.21.15.12:0/4209094071 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7fe7f4051c20 msgr2=0x7fe7f40919b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=0 2023-12-03T11:38:16.264 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=1 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=2 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=3 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=4 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=5 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.263+0000 7fe803922800 20 add_watcher() i=6 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.264+0000 7fe803922800 20 add_watcher() i=7 2023-12-03T11:38:16.265 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.264+0000 7fe803922800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.267 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.266+0000 7fe7c57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.267+0000 7fe803922800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.268 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.267+0000 7fe803922800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.272+0000 7fe803922800 5 note: GC not initialized 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe733fff640 20 reqs_thread_entry: start 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe803922800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe803922800 20 Filter name: none 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe731ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:16.274 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.273+0000 7fe803922800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 INFO: main: starting shard=0 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 INFO: main: starting shard=1 2023-12-03T11:38:16.276 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.274+0000 7fe803922800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 20 INFO: main: starting shard=2 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.275+0000 7fe803922800 20 INFO: main: done 2023-12-03T11:38:16.277 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.276+0000 7fe7c57e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.278+0000 7fe803922800 20 remove_watcher() i=0 2023-12-03T11:38:16.279 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.278+0000 7fe803922800 2 removed watcher, disabling cache 2023-12-03T11:38:16.280 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.279+0000 7fe803922800 20 remove_watcher() i=1 2023-12-03T11:38:16.281 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.280+0000 7fe803922800 20 remove_watcher() i=2 2023-12-03T11:38:16.282 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.281+0000 7fe803922800 20 remove_watcher() i=3 2023-12-03T11:38:16.283 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.282+0000 7fe803922800 20 remove_watcher() i=4 2023-12-03T11:38:16.284 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.283+0000 7fe803922800 20 remove_watcher() i=5 2023-12-03T11:38:16.285 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.284+0000 7fe803922800 20 remove_watcher() i=6 2023-12-03T11:38:16.286 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.285+0000 7fe803922800 20 remove_watcher() i=7 2023-12-03T11:38:16.293 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket a-bucket --num-shards 1 --yes-i-really-mean-it 2023-12-03T11:38:16.323 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.324 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.338 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.337+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 realm 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.339+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.340+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.340+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.341+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.341+0000 7f58c593a800 20 searching for the correct realm 2023-12-03T11:38:16.552 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.551+0000 7f58c1e0e640 0 -- 172.21.15.12:0/2135522872 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x565097b176c0 msgr2=0x565097b19ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.553+0000 7f58c593a800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.553+0000 7f58c593a800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.553+0000 7f58c593a800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.553+0000 7f58c593a800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.553+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.555 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 zone default found 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 4 Realm: () 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 zonegroup default 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.554+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.555+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.555+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.555+0000 7f58c593a800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.555+0000 7f58c593a800 20 started sync module instance, tier type = 2023-12-03T11:38:16.557 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.555+0000 7f58c593a800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.560 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.559+0000 7f58c0e0c640 0 -- 172.21.15.12:0/2135522872 >> [v2:172.21.15.191:6824/2936799433,v1:172.21.15.191:6825/2936799433] conn(0x565097b1eef0 msgr2=0x565097b212e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=0 2023-12-03T11:38:16.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=1 2023-12-03T11:38:16.565 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=2 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=3 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=4 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=5 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.564+0000 7f58c593a800 20 add_watcher() i=6 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.565+0000 7f58c593a800 20 add_watcher() i=7 2023-12-03T11:38:16.566 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.565+0000 7f58c593a800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.567+0000 7f5886fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.567+0000 7f58c593a800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.568 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.567+0000 7f58c593a800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.573+0000 7f58c593a800 5 note: GC not initialized 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.573+0000 7f57f5ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.573+0000 7f58c593a800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 20 Filter name: none 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f57e77fe640 20 reqs_thread_entry: start 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 10 cache get: name=default.rgw.meta+root+a-bucket : miss 2023-12-03T11:38:16.575 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 10 cache put: name=default.rgw.meta+root+a-bucket info.flags=0x11 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 10 adding default.rgw.meta+root+a-bucket to cache LRU end 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : miss 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.574+0000 7f58c593a800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.576 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:38:16.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 10 adding default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:38:16.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:16.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.577 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.575+0000 7f58c593a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:38:16.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58c593a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:38:16.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58c593a800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:38:16.580 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58c593a800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58c593a800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58a2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725947 cookie 94904142402320 notifier 7473 bl.length()=789 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58a2ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58a2ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.579+0000 7f58a2ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.580+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.580+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.580+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:38:16.581 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.580+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:38:16.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.3.0 2023-12-03T11:38:16.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.3.0 2023-12-03T11:38:16.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:38:16.582 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.0 2023-12-03T11:38:16.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 10 Dropping entry with empty name, idx=�1001_04/57/0457f727ec113e418d5b16d206b200ed068c0533554883ce811df7c932a3df68/2018_12_11/2889999/3386469/metadata.gz 2023-12-03T11:38:16.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:38:16.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.581+0000 7f58c593a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.1 2023-12-03T11:38:16.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.582+0000 7f58c593a800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:38:16.583 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.582+0000 7f58c593a800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.2.2 2023-12-03T11:38:16.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.582+0000 7f58c593a800 10 cache get: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 : hit (requested=0x17, cached=0x17) 2023-12-03T11:38:16.584 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.582+0000 7f58c593a800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.583+0000 7f58c593a800 10 cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.583+0000 7f58c593a800 10 moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.583+0000 7f58c593a800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.583+0000 7f58c593a800 10 distributing notification oid=default.rgw.control:notify.5 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16, ofs0, ns] 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58a2ffd640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725948 cookie 94904142402320 notifier 7473 bl.length()=757 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58a2ffd640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 info.flags=0x17 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58a2ffd640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.a-bucket:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16 to cache LRU end 2023-12-03T11:38:16.585 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58a2ffd640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_sources= new_sources= 2023-12-03T11:38:16.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential sources added=[] removed=[] 2023-12-03T11:38:16.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): orig_dests= new_dests= 2023-12-03T11:38:16.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.584+0000 7f58c593a800 20 handle_bi_update(): bucket=:a-bucket[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16]): potential dests added=[] removed=[] 2023-12-03T11:38:16.586 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.585+0000 7f58c593a800 1 execute INFO: reshard of bucket "a-bucket" completed successfully 2023-12-03T11:38:16.588 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.587+0000 7f5886fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.589+0000 7f58c593a800 20 remove_watcher() i=0 2023-12-03T11:38:16.590 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.589+0000 7f58c593a800 2 removed watcher, disabling cache 2023-12-03T11:38:16.591 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.590+0000 7f58c593a800 20 remove_watcher() i=1 2023-12-03T11:38:16.592 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.591+0000 7f58c593a800 20 remove_watcher() i=2 2023-12-03T11:38:16.593 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.592+0000 7f58c593a800 20 remove_watcher() i=3 2023-12-03T11:38:16.594 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.593+0000 7f58c593a800 20 remove_watcher() i=4 2023-12-03T11:38:16.595 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.594+0000 7f58c593a800 20 remove_watcher() i=5 2023-12-03T11:38:16.596 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.595+0000 7f58c593a800 20 remove_watcher() i=6 2023-12-03T11:38:16.597 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.596+0000 7f58c593a800 20 remove_watcher() i=7 2023-12-03T11:38:16.603 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket a-bucket 2023-12-03T11:38:16.631 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.631 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.645 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.644+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 realm 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.647 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.646+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.647+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.648 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.647+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.648+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.649 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.648+0000 7fc65e612800 20 searching for the correct realm 2023-12-03T11:38:16.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65a52b640 0 -- 172.21.15.12:0/2751635653 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7fc650028910 msgr2=0x7fc65001e400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65e612800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65e612800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65e612800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65e612800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.658+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 zone default found 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 4 Realm: () 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 zonegroup default 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.659+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.660+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.660+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.660+0000 7fc65e612800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.660+0000 7fc65e612800 20 started sync module instance, tier type = 2023-12-03T11:38:16.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.660+0000 7fc65e612800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.669 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.668+0000 7fc65a52b640 0 -- 172.21.15.12:0/2751635653 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5644b8085440 msgr2=0x5644b8087890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=0 2023-12-03T11:38:16.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=1 2023-12-03T11:38:16.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=2 2023-12-03T11:38:16.670 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=3 2023-12-03T11:38:16.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=4 2023-12-03T11:38:16.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.669+0000 7fc65e612800 20 add_watcher() i=5 2023-12-03T11:38:16.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.670+0000 7fc65e612800 20 add_watcher() i=6 2023-12-03T11:38:16.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.670+0000 7fc65e612800 20 add_watcher() i=7 2023-12-03T11:38:16.671 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.670+0000 7fc65e612800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.672+0000 7fc61f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.672+0000 7fc65e612800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.673 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.672+0000 7fc65e612800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.678+0000 7fc65e612800 5 note: GC not initialized 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.678+0000 7fc58e7fc640 20 reqs_thread_entry: start 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.678+0000 7fc65e612800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.679+0000 7fc65e612800 20 Filter name: none 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.679+0000 7fc57ffff640 20 reqs_thread_entry: start 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.679+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.680 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.679+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=176 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.679+0000 7fc65e612800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 20 rados_obj.operate() r=0 bl.length=360 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 20 INFO: main: max_entries=1000, index=gen=3, max_shards=1 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 20 INFO: main: starting shard=0 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.3.0 2023-12-03T11:38:16.681 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.680+0000 7fc65e612800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16.3.0 2023-12-03T11:38:16.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.681+0000 7fc65e612800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:16.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.681+0000 7fc65e612800 20 INFO: main: done 2023-12-03T11:38:16.682 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.681+0000 7fc61f7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.683+0000 7fc65e612800 20 remove_watcher() i=0 2023-12-03T11:38:16.684 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.683+0000 7fc65e612800 2 removed watcher, disabling cache 2023-12-03T11:38:16.685 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.684+0000 7fc65e612800 20 remove_watcher() i=1 2023-12-03T11:38:16.687 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.685+0000 7fc65e612800 20 remove_watcher() i=2 2023-12-03T11:38:16.688 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.687+0000 7fc65e612800 20 remove_watcher() i=3 2023-12-03T11:38:16.689 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.688+0000 7fc65e612800 20 remove_watcher() i=4 2023-12-03T11:38:16.690 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.689+0000 7fc65e612800 20 remove_watcher() i=5 2023-12-03T11:38:16.691 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.690+0000 7fc65e612800 20 remove_watcher() i=6 2023-12-03T11:38:16.692 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.691+0000 7fc65e612800 20 remove_watcher() i=7 2023-12-03T11:38:16.696 INFO:tasks.workunit.client.0.smithi012.stderr: test: PUT succeeds during reshard 2023-12-03T11:38:16.698 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:38:16.726 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.726 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.740 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.739+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.740+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.740+0000 7fa54f733800 20 realm 2023-12-03T11:38:16.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.740+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.741+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.742 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.741+0000 7fa54f733800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.741+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.741+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.741+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.742+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.743 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.742+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.743+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.744 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.743+0000 7fa54f733800 20 searching for the correct realm 2023-12-03T11:38:16.753 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.752+0000 7fa54b468640 0 -- 172.21.15.12:0/2689741796 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x56160b23de80 msgr2=0x56160b240270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.754 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.755 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.753+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 zone default found 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 4 Realm: () 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.756 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 zonegroup default 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.754+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.755+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.755+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.755+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.755+0000 7fa54f733800 20 started sync module instance, tier type = 2023-12-03T11:38:16.757 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.755+0000 7fa54f733800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.761 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.760+0000 7fa54b468640 0 -- 172.21.15.12:0/2689741796 >> [v2:172.21.15.12:6808/3792383689,v1:172.21.15.12:6809/3792383689] conn(0x7fa534059a10 msgr2=0x7fa53405aa30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.764 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=0 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=1 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=2 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=3 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=4 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=5 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.763+0000 7fa54f733800 20 add_watcher() i=6 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.764+0000 7fa54f733800 20 add_watcher() i=7 2023-12-03T11:38:16.765 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.764+0000 7fa54f733800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.766+0000 7fa5117e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.766+0000 7fa54f733800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.767 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.766+0000 7fa54f733800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.772+0000 7fa54f733800 5 note: GC not initialized 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.772+0000 7fa47ffff640 20 reqs_thread_entry: start 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa54f733800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa54f733800 20 Filter name: none 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa47dffb640 20 reqs_thread_entry: start 2023-12-03T11:38:16.774 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.773+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.775 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.774+0000 7fa54f733800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.775+0000 7fa54f733800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.776 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.775+0000 7fa54f733800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.777 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.776+0000 7fa5117e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.778+0000 7fa54f733800 20 remove_watcher() i=0 2023-12-03T11:38:16.779 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.778+0000 7fa54f733800 2 removed watcher, disabling cache 2023-12-03T11:38:16.780 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.779+0000 7fa54f733800 20 remove_watcher() i=1 2023-12-03T11:38:16.781 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.780+0000 7fa54f733800 20 remove_watcher() i=2 2023-12-03T11:38:16.782 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.781+0000 7fa54f733800 20 remove_watcher() i=3 2023-12-03T11:38:16.783 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.782+0000 7fa54f733800 20 remove_watcher() i=4 2023-12-03T11:38:16.784 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.783+0000 7fa54f733800 20 remove_watcher() i=5 2023-12-03T11:38:16.785 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.784+0000 7fa54f733800 20 remove_watcher() i=6 2023-12-03T11:38:16.786 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.785+0000 7fa54f733800 20 remove_watcher() i=7 2023-12-03T11:38:16.791 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:16.791 INFO:tasks.workunit.client.0.smithi012.stderr:BUCKET_STATS: 2023-12-03T11:38:16.791 INFO:tasks.workunit.client.0.smithi012.stderr:bucket: myver id: ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 num_objects: 0 size_kb: 0 num_shards: 2 2023-12-03T11:38:16.791 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:16.792 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin --inject-delay-at=do_reshard --inject-delay-ms=5000 bucket reshard --bucket myver --num-shards 3 2023-12-03T11:38:16.824 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:16.824 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:16.839 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.838+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 20 realm 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.839+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.840+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.841 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.840+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.841+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.842 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.841+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.842+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:16.843 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.842+0000 7f4a3713b800 20 searching for the correct realm 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a33641640 0 -- 172.21.15.12:0/449313487 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7f4a28029410 msgr2=0x7f4a28029890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a3713b800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a3713b800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a3713b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a3713b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.852+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.855 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 zone default found 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 4 Realm: () 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 zonegroup default 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.853+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.854+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.856 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.854+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.854+0000 7f4a3713b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:16.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.854+0000 7f4a3713b800 20 started sync module instance, tier type = 2023-12-03T11:38:16.857 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.854+0000 7f4a3713b800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:16.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a33641640 0 -- 172.21.15.12:0/449313487 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x56462e2c8710 msgr2=0x56462e2cabd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:16.864 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=0 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=1 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=2 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=3 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=4 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=5 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.863+0000 7f4a3713b800 20 add_watcher() i=6 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.864+0000 7f4a3713b800 20 add_watcher() i=7 2023-12-03T11:38:16.865 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.864+0000 7f4a3713b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.866+0000 7f4973fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.866+0000 7f4a3713b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:16.868 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.866+0000 7f4a3713b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:16.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4a3713b800 5 note: GC not initialized 2023-12-03T11:38:16.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4962ffd640 20 reqs_thread_entry: start 2023-12-03T11:38:16.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4a3713b800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:16.874 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4a3713b800 20 Filter name: none 2023-12-03T11:38:16.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4960ff9640 20 reqs_thread_entry: start 2023-12-03T11:38:16.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4a3713b800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:38:16.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.873+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.875 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : miss 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 20 rados->read ofs=0 len=0 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 adding default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.876 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.874+0000 7f4a3713b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:16.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.879+0000 7f4a3713b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:16.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.879+0000 7f4a3713b800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:16.881 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.879+0000 7f4a3713b800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.879+0000 7f4a3713b800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.879+0000 7f4a18ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725949 cookie 94859422409744 notifier 7539 bl.length()=783 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a18ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a18ff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a18ff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:16.882 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.880+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:16.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:16.881+0000 7f4a3713b800 -1 Injecting delay=5.000000000s at location=do_reshard 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.0 2023-12-03T11:38:21.883 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.0 2023-12-03T11:38:21.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.1 2023-12-03T11:38:21.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.882+0000 7f4a3713b800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.1.1 2023-12-03T11:38:21.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.883+0000 7f4a3713b800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:38:21.884 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.883+0000 7f4a3713b800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a3713b800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a3713b800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a3713b800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a3713b800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a18ff9640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725950 cookie 94859422409744 notifier 7539 bl.length()=751 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a18ff9640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:21.886 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a18ff9640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:21.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.885+0000 7f4a18ff9640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:21.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.886+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:21.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.886+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:21.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.886+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:21.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.886+0000 7f4a3713b800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:21.889 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.888+0000 7f4a3713b800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:38:21.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.890+0000 7f4973fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:21.894 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.893+0000 7f4a3713b800 20 remove_watcher() i=0 2023-12-03T11:38:21.895 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.893+0000 7f4a3713b800 2 removed watcher, disabling cache 2023-12-03T11:38:21.896 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.894+0000 7f4a3713b800 20 remove_watcher() i=1 2023-12-03T11:38:21.897 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.896+0000 7f4a3713b800 20 remove_watcher() i=2 2023-12-03T11:38:21.898 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.897+0000 7f4a3713b800 20 remove_watcher() i=3 2023-12-03T11:38:21.900 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.899+0000 7f4a3713b800 20 remove_watcher() i=4 2023-12-03T11:38:21.901 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.900+0000 7f4a3713b800 20 remove_watcher() i=5 2023-12-03T11:38:21.902 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.901+0000 7f4a3713b800 20 remove_watcher() i=6 2023-12-03T11:38:21.903 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:21.902+0000 7f4a3713b800 20 remove_watcher() i=7 2023-12-03T11:38:22.824 INFO:tasks.workunit.client.0.smithi012.stderr:put object successful 2023-12-03T11:38:22.824 INFO:tasks.workunit.client.0.smithi012.stderr:TEST: check that bucket stats are correct after reshard with unlinked entries 2023-12-03T11:38:22.825 INFO:tasks.workunit.client.0.smithi012.stderr: 2023-12-03T11:38:22.977 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_set_olh_err 2 2023-12-03T11:38:23.381 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config set client rgw_debug_inject_olh_cancel_modification_err true 2023-12-03T11:38:24.834 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:38:24.835 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket myver --object x 2023-12-03T11:38:24.871 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:24.871 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:24.887 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.885+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 20 realm 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.887+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.888+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.890 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.888+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.889+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:24.891 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.889+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.890+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:24.892 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.890+0000 7f2c37f11800 20 searching for the correct realm 2023-12-03T11:38:24.904 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.902+0000 7f2c36212640 0 -- 172.21.15.12:0/4192014710 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55c9e876f300 msgr2=0x55c9e87717c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.905 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.906 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.903+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 zone default found 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 4 Realm: () 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:24.907 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 zonegroup default 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.904+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.905+0000 7f2c37f11800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.905+0000 7f2c37f11800 20 started sync module instance, tier type = 2023-12-03T11:38:24.908 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.905+0000 7f2c37f11800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:24.915 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.913+0000 7f2c36212640 0 -- 172.21.15.12:0/4192014710 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x7f2c10079db0 msgr2=0x7f2c1007c1e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=0 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=1 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=2 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=3 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=4 2023-12-03T11:38:24.918 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=5 2023-12-03T11:38:24.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.916+0000 7f2c37f11800 20 add_watcher() i=6 2023-12-03T11:38:24.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.917+0000 7f2c37f11800 20 add_watcher() i=7 2023-12-03T11:38:24.919 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.917+0000 7f2c37f11800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:24.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.919+0000 7f2bfb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:24.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.919+0000 7f2c37f11800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:24.921 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.919+0000 7f2c37f11800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.925+0000 7f2c37f11800 5 note: GC not initialized 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2b6a7fc640 20 reqs_thread_entry: start 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2c37f11800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2c37f11800 20 Filter name: none 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2b5bfff640 20 reqs_thread_entry: start 2023-12-03T11:38:24.928 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:24.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.926+0000 7f2c37f11800 20 rados->read ofs=0 len=0 2023-12-03T11:38:24.929 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:24.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:24.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2023-12-03T11:38:24.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 20 INFO: main: starting shard=0 2023-12-03T11:38:24.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:24.930 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.927+0000 7f2c37f11800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.928+0000 7f2c37f11800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.928+0000 7f2c37f11800 20 INFO: main: starting shard=1 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.928+0000 7f2c37f11800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.928+0000 7f2c37f11800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 20 INFO: main: starting shard=2 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:24.931 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:24.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.929+0000 7f2c37f11800 20 INFO: main: done 2023-12-03T11:38:24.932 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.930+0000 7f2bfb7e6640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:24.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.933+0000 7f2c37f11800 20 remove_watcher() i=0 2023-12-03T11:38:24.935 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.933+0000 7f2c37f11800 2 removed watcher, disabling cache 2023-12-03T11:38:24.936 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.934+0000 7f2c37f11800 20 remove_watcher() i=1 2023-12-03T11:38:24.937 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.935+0000 7f2c37f11800 20 remove_watcher() i=2 2023-12-03T11:38:24.938 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.936+0000 7f2c37f11800 20 remove_watcher() i=3 2023-12-03T11:38:24.939 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.937+0000 7f2c37f11800 20 remove_watcher() i=4 2023-12-03T11:38:24.940 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.938+0000 7f2c37f11800 20 remove_watcher() i=5 2023-12-03T11:38:24.941 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.939+0000 7f2c37f11800 20 remove_watcher() i=6 2023-12-03T11:38:24.942 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:24.940+0000 7f2c37f11800 20 remove_watcher() i=7 2023-12-03T11:38:25.006 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:38:25.007 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket myver --object y 2023-12-03T11:38:25.038 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:25.038 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:25.052 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.050+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 realm 2023-12-03T11:38:25.054 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.052+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.053+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:25.055 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.053+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.054+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:25.056 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.054+0000 7faca3b31800 20 searching for the correct realm 2023-12-03T11:38:25.067 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.065+0000 7fac9f868640 0 -- 172.21.15.12:0/3397140829 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x5611330c3430 msgr2=0x5611330c5820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.068 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:25.069 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.066+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 zone default found 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 4 Realm: () 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 zonegroup default 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.070 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.067+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.068+0000 7faca3b31800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.068+0000 7faca3b31800 20 started sync module instance, tier type = 2023-12-03T11:38:25.071 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.068+0000 7faca3b31800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:25.078 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.076+0000 7faca0069640 0 -- 172.21.15.12:0/3397140829 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x5611330cb2c0 msgr2=0x5611330cd780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:25.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=0 2023-12-03T11:38:25.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=1 2023-12-03T11:38:25.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=2 2023-12-03T11:38:25.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=3 2023-12-03T11:38:25.079 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=4 2023-12-03T11:38:25.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.077+0000 7faca3b31800 20 add_watcher() i=5 2023-12-03T11:38:25.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.078+0000 7faca3b31800 20 add_watcher() i=6 2023-12-03T11:38:25.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.078+0000 7faca3b31800 20 add_watcher() i=7 2023-12-03T11:38:25.080 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.078+0000 7faca3b31800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:25.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.080+0000 7fac65fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:25.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.080+0000 7faca3b31800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:25.082 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.080+0000 7faca3b31800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.086+0000 7faca3b31800 5 note: GC not initialized 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.086+0000 7fabdcff9640 20 reqs_thread_entry: start 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.086+0000 7faca3b31800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 Filter name: none 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7fabd27fc640 20 reqs_thread_entry: start 2023-12-03T11:38:25.089 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 INFO: main: starting shard=0 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:25.090 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.087+0000 7faca3b31800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:25.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.088+0000 7faca3b31800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:25.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.088+0000 7faca3b31800 20 INFO: main: starting shard=1 2023-12-03T11:38:25.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.088+0000 7faca3b31800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:25.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.088+0000 7faca3b31800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:25.091 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 20 INFO: main: starting shard=2 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.089+0000 7faca3b31800 20 INFO: main: done 2023-12-03T11:38:25.092 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.090+0000 7fac65fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:25.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.092+0000 7faca3b31800 20 remove_watcher() i=0 2023-12-03T11:38:25.094 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.092+0000 7faca3b31800 2 removed watcher, disabling cache 2023-12-03T11:38:25.096 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.094+0000 7faca3b31800 20 remove_watcher() i=1 2023-12-03T11:38:25.097 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.095+0000 7faca3b31800 20 remove_watcher() i=2 2023-12-03T11:38:25.098 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.096+0000 7faca3b31800 20 remove_watcher() i=3 2023-12-03T11:38:25.099 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.097+0000 7faca3b31800 20 remove_watcher() i=4 2023-12-03T11:38:25.100 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.098+0000 7faca3b31800 20 remove_watcher() i=5 2023-12-03T11:38:25.101 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.099+0000 7faca3b31800 20 remove_watcher() i=6 2023-12-03T11:38:25.102 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.100+0000 7faca3b31800 20 remove_watcher() i=7 2023-12-03T11:38:25.164 INFO:tasks.workunit.client.0.smithi012.stderr:An error occurred (NoSuchKey) when calling the PutObject operation: None 2023-12-03T11:38:25.166 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bi list --bucket myver --object z 2023-12-03T11:38:25.195 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:25.196 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:25.210 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.207+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.211 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 realm 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.212 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.209+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.211+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:25.213 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.211+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.212+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:25.214 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.212+0000 7f9553927800 20 searching for the correct realm 2023-12-03T11:38:25.224 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f954f6bf640 0 -- 172.21.15.12:0/1029818141 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x563b0836b690 msgr2=0x563b0836da80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.225 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.222+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:25.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.226 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 zone default found 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 4 Realm: () 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 zonegroup default 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.223+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.224+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.224+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.224+0000 7f9553927800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.224+0000 7f9553927800 20 started sync module instance, tier type = 2023-12-03T11:38:25.227 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.224+0000 7f9553927800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:25.234 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.232+0000 7f954fec0640 0 -- 172.21.15.12:0/1029818141 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x563b083733f0 msgr2=0x563b08375890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=0 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=1 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=2 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=3 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=4 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=5 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.233+0000 7f9553927800 20 add_watcher() i=6 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.234+0000 7f9553927800 20 add_watcher() i=7 2023-12-03T11:38:25.236 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.234+0000 7f9553927800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:25.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.236+0000 7f95157e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:25.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.236+0000 7f9553927800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:25.238 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.236+0000 7f9553927800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.242+0000 7f9553927800 5 note: GC not initialized 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.242+0000 7f9483fff640 20 reqs_thread_entry: start 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9553927800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9553927800 20 Filter name: none 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9481ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:25.245 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.243+0000 7f9553927800 20 rados->read ofs=0 len=0 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 INFO: main: max_entries=1000, index=gen=2, max_shards=3 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 INFO: main: starting shard=0 2023-12-03T11:38:25.246 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 INFO: main: starting shard=1 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.244+0000 7f9553927800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 20 INFO: main: bi_list() returned without error; entries.size()=1, is_truncated=0, marker= 2023-12-03T11:38:25.247 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 20 INFO: main: starting shard=2 2023-12-03T11:38:25.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:25.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:25.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 20 INFO: main: bi_list() returned without error; entries.size()=0, is_truncated=0, marker= 2023-12-03T11:38:25.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.245+0000 7f9553927800 20 INFO: main: done 2023-12-03T11:38:25.248 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.246+0000 7f95157e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:25.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.248+0000 7f9553927800 20 remove_watcher() i=0 2023-12-03T11:38:25.251 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.248+0000 7f9553927800 2 removed watcher, disabling cache 2023-12-03T11:38:25.252 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.250+0000 7f9553927800 20 remove_watcher() i=1 2023-12-03T11:38:25.253 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.251+0000 7f9553927800 20 remove_watcher() i=2 2023-12-03T11:38:25.254 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.252+0000 7f9553927800 20 remove_watcher() i=3 2023-12-03T11:38:25.255 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.253+0000 7f9553927800 20 remove_watcher() i=4 2023-12-03T11:38:25.256 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.254+0000 7f9553927800 20 remove_watcher() i=5 2023-12-03T11:38:25.257 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.255+0000 7f9553927800 20 remove_watcher() i=6 2023-12-03T11:38:25.258 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:25.256+0000 7f9553927800 20 remove_watcher() i=7 2023-12-03T11:38:25.269 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_set_olh_err 2023-12-03T11:38:25.662 INFO:tasks.workunit.client.0.smithi012.stderr:ceph config rm client rgw_debug_inject_olh_cancel_modification_err 2023-12-03T11:38:26.054 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket reshard --bucket myver --num-shards 17 --yes-i-really-mean-it 2023-12-03T11:38:26.086 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:26.086 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:26.298 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.296+0000 7f57b50f0640 0 --2- 172.21.15.12:0/1758926972 >> [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] conn(0x564ed6460170 0x564ed6461640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-12-03T11:38:26.302 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.300+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.304 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 realm 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.305 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.302+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.303+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.303+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.304+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:26.306 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.304+0000 7f57b932c800 20 searching for the correct realm 2023-12-03T11:38:26.315 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.313+0000 7f57b50f0640 0 -- 172.21.15.12:0/3473699934 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x564ed6481480 msgr2=0x564ed6483870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:38:26.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:26.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:26.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:26.319 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.316+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 zone default found 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 4 Realm: () 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:26.320 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 zonegroup default 2023-12-03T11:38:26.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.321 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.317+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.318+0000 7f57b932c800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.318+0000 7f57b932c800 20 started sync module instance, tier type = 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.318+0000 7f57b932c800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:26.322 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.319+0000 7f57b50f0640 0 -- 172.21.15.12:0/3473699934 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x7f579007a9d0 msgr2=0x7f579007ce00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:38:26.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=0 2023-12-03T11:38:26.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=1 2023-12-03T11:38:26.330 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=2 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=3 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=4 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=5 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.328+0000 7f57b932c800 20 add_watcher() i=6 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.329+0000 7f57b932c800 20 add_watcher() i=7 2023-12-03T11:38:26.331 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.329+0000 7f57b932c800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:26.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.331+0000 7f577afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.331+0000 7f57b932c800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:26.333 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.331+0000 7f57b932c800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.337+0000 7f57b932c800 5 note: GC not initialized 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.337+0000 7f56e9ffb640 20 reqs_thread_entry: start 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.337+0000 7f57b932c800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 20 Filter name: none 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f56db7fe640 20 reqs_thread_entry: start 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:38:26.340 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:26.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:38:26.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:38:26.341 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : miss 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 adding default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.342 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.338+0000 7f57b932c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:26.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57b932c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57b932c800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.347 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57b932c800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57b932c800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57977fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725951 cookie 94896602546304 notifier 7244 bl.length()=783 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57977fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57977fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.345+0000 7f57977fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.346+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.346+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.346+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:26.348 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.346+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.0 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.0 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.1 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.1 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.2 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.2 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.3 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.3 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.4 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.4 2023-12-03T11:38:26.350 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.5 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.5 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.6 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.6 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.7 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.7 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.8 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.8 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.10 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.10 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.11 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.11 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.12 2023-12-03T11:38:26.351 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.12 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.13 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.13 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.14 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.14 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.15 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.15 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.16 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.16 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.347+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.0 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.348+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:26.352 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.348+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.1 2023-12-03T11:38:26.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.348+0000 7f57b932c800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:26.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.348+0000 7f57b932c800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.2.2 2023-12-03T11:38:26.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.350+0000 7f57b932c800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : hit (requested=0x17, cached=0x17) 2023-12-03T11:38:26.353 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.350+0000 7f57b932c800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 10 moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:root:.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17, ofs0, ns] 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57977fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725952 cookie 94896602546304 notifier 7244 bl.length()=751 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57977fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57977fe640 10 rgw watcher librados: moving default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.354 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57977fe640 10 rgw watcher librados: updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_sources= new_sources= 2023-12-03T11:38:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential sources added=[] removed=[] 2023-12-03T11:38:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): orig_dests= new_dests= 2023-12-03T11:38:26.355 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.352+0000 7f57b932c800 20 handle_bi_update(): bucket=:myver[ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17]): potential dests added=[] removed=[] 2023-12-03T11:38:26.356 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.354+0000 7f57b932c800 1 execute INFO: reshard of bucket "myver" completed successfully 2023-12-03T11:38:26.357 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.355+0000 7f577afe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.358+0000 7f57b932c800 20 remove_watcher() i=0 2023-12-03T11:38:26.360 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.358+0000 7f57b932c800 2 removed watcher, disabling cache 2023-12-03T11:38:26.361 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.359+0000 7f57b932c800 20 remove_watcher() i=1 2023-12-03T11:38:26.362 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.360+0000 7f57b932c800 20 remove_watcher() i=2 2023-12-03T11:38:26.363 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.361+0000 7f57b932c800 20 remove_watcher() i=3 2023-12-03T11:38:26.364 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.362+0000 7f57b932c800 20 remove_watcher() i=4 2023-12-03T11:38:26.365 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.363+0000 7f57b932c800 20 remove_watcher() i=5 2023-12-03T11:38:26.366 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.364+0000 7f57b932c800 20 remove_watcher() i=6 2023-12-03T11:38:26.367 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.365+0000 7f57b932c800 20 remove_watcher() i=7 2023-12-03T11:38:26.373 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket check unlinked --bucket myver --fix --min-age-hours 0 --rgw-olh-pending-timeout-sec 0 --dump-keys 2023-12-03T11:38:26.407 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:26.408 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:26.422 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.419+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 realm 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.424 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.421+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.422+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.422+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.425 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.423+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:26.426 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.423+0000 7f98af72d800 20 searching for the correct realm 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98abb7f640 0 -- 172.21.15.12:0/840289903 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x561ae4fa8000 msgr2=0x561ae4faa3f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:26.438 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.435+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 zone default found 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 4 Realm: () 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 zonegroup default 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.439 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 started sync module instance, tier type = 2023-12-03T11:38:26.440 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.436+0000 7f98af72d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98ab37e640 0 -- 172.21.15.12:0/840289903 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f98a006f920 msgr2=0x7f98a005d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=0 2023-12-03T11:38:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=1 2023-12-03T11:38:26.448 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=2 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=3 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=4 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=5 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.446+0000 7f98af72d800 20 add_watcher() i=6 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.447+0000 7f98af72d800 20 add_watcher() i=7 2023-12-03T11:38:26.449 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.447+0000 7f98af72d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.449+0000 7f98717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.449+0000 7f98af72d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:26.451 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.449+0000 7f98af72d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.455+0000 7f98af72d800 5 note: GC not initialized 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.455+0000 7f97dffff640 20 reqs_thread_entry: start 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.455+0000 7f98af72d800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 20 Filter name: none 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f97ddffb640 20 reqs_thread_entry: start 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 10 cache get: name=default.rgw.meta+root+myver : miss 2023-12-03T11:38:26.458 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:26.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 10 cache put: name=default.rgw.meta+root+myver info.flags=0x11 2023-12-03T11:38:26.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 10 adding default.rgw.meta+root+myver to cache LRU end 2023-12-03T11:38:26.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 10 cache get: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 : miss 2023-12-03T11:38:26.459 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.456+0000 7f98af72d800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 cache put: name=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 info.flags=0x17 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 adding default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 to cache LRU end 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 updating xattr: name=ceph.objclass.version bl.length()=42 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 updating xattr: name=user.rgw.acl bl.length()=143 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 chain_cache_entry: cache_locator=default.rgw.meta+root+.bucket.meta.myver:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.0 2023-12-03T11:38:26.460 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.0 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 1 NOTICE: finished shard 0 (0 entries removed) 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.1 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.457+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.1 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 1 NOTICE: finished shard 1 (0 entries removed) 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.2 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.2 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 1 NOTICE: finished shard 2 (0 entries removed) 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.3 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.458+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.3 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 1 NOTICE: finished shard 3 (0 entries removed) 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.4 2023-12-03T11:38:26.461 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.4 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 1 NOTICE: finished shard 4 (0 entries removed) 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.5 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.459+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.5 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 1 NOTICE: finished shard 5 (0 entries removed) 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.6 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.6 2023-12-03T11:38:26.462 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 1 NOTICE: finished shard 6 (0 entries removed) 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.7 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.7 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 1 NOTICE: finished shard 7 (0 entries removed) 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.8 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.460+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.8 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.461+0000 7f98af72d800 1 NOTICE: finished shard 8 (0 entries removed) 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.461+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.463 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.461+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.10 2023-12-03T11:38:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.10 2023-12-03T11:38:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 1 NOTICE: finished shard 10 (0 entries removed) 2023-12-03T11:38:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.11 2023-12-03T11:38:26.464 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.11 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 1 NOTICE: finished shard 11 (0 entries removed) 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.12 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.462+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.12 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 1 NOTICE: finished shard 12 (0 entries removed) 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.13 2023-12-03T11:38:26.465 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.13 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 1 NOTICE: finished shard 13 (0 entries removed) 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.14 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.463+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.14 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 1 NOTICE: finished shard 14 (0 entries removed) 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.15 2023-12-03T11:38:26.466 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.15 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 1 NOTICE: finished shard 15 (0 entries removed) 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 10 bucket_obj is .dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.16 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.16 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.464+0000 7f98af72d800 1 NOTICE: finished shard 16 (0 entries removed) 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.465+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.467 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.465+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:x state=0x561ae5061bb8 s->prefetch_data=0 2023-12-03T11:38:26.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.465+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to n7pgyoe1grpf618wzp9fdmu82slh0kjq 2023-12-03T11:38:26.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.465+0000 7f98af72d800 20 get_obj_state_impl: setting s->olh_tag to fpiepj2k1o5ceg58398ln53litzj0age 2023-12-03T11:38:26.468 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.465+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=fpiepj2k1o5ceg58398ln53litzj0age 2023-12-03T11:38:26.469 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.467+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.468+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.470 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.468+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=2 key=x[] 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.468+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=3 key=x[pwQ6Sl5nT4LQDGWmssIcmhWOKoDDVzx] 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.468+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:pwQ6Sl5nT4LQDGWmssIcmhWOKoDDVzx_x state=0x561ae506e688 s->prefetch_data=0 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.469+0000 7f98af72d800 10 manifest: total_size = 22 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.469+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2563902367259067766 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.469+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:pwQ6Sl5nT4LQDGWmssIcmhWOKoDDVzx_x state=0x561ae506e688 s->prefetch_data=0 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.469+0000 7f98af72d800 20 prepare_atomic_modification: state is not atomic. state=0x561ae506e688 2023-12-03T11:38:26.471 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.469+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.472+0000 7f98af72d800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:38:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.472+0000 7f98af72d800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:38:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.472+0000 7f98af72d800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:38:26.474 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.472+0000 7f98af72d800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:pwQ6Sl5nT4LQDGWmssIcmhWOKoDDVzx_.eeLnlZgfH2WUy3SQV125vikVuniUXVT_0 2023-12-03T11:38:26.475 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.473+0000 7f9891ffb640 20 handle_completion(): completion ok for obj=x[pwQ6Sl5nT4LQDGWmssIcmhWOKoDDVzx] 2023-12-03T11:38:26.476 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.474+0000 7f98af72d800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:38:26.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.475+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:x state=0x561ae5061bb8 s->prefetch_data=0 2023-12-03T11:38:26.477 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.475+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=fpiepj2k1o5ceg58398ln53litzj0age 2023-12-03T11:38:26.480 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.478+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.480+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.480+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.482 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.480+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:y state=0x561ae5061b88 s->prefetch_data=0 2023-12-03T11:38:26.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.481+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to uhgscinav01a0h2pbv0nj3f3496a82d0 2023-12-03T11:38:26.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.481+0000 7f98af72d800 20 get_obj_state_impl: setting s->olh_tag to 3fzpatogphxvg3h95xfnnd4kyyv8zoho 2023-12-03T11:38:26.483 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.481+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=3fzpatogphxvg3h95xfnnd4kyyv8zoho 2023-12-03T11:38:26.484 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.482+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.485 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.483+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=2 key=y[] 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=3 key=y[2I4xYJPrijm.f9mSMlciZFsjxofDkV9] 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:2I4xYJPrijm.f9mSMlciZFsjxofDkV9_y state=0x561ae50695e8 s->prefetch_data=0 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 10 manifest: total_size = 12 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12335112530832609587 2023-12-03T11:38:26.486 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:2I4xYJPrijm.f9mSMlciZFsjxofDkV9_y state=0x561ae50695e8 s->prefetch_data=0 2023-12-03T11:38:26.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 prepare_atomic_modification: state is not atomic. state=0x561ae50695e8 2023-12-03T11:38:26.487 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.484+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.487+0000 7f98af72d800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:38:26.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.487+0000 7f98af72d800 20 RGWObjManifest::operator++(): result: ofs=12 stripe_ofs=12 part_ofs=0 rule->part_size=0 2023-12-03T11:38:26.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.487+0000 7f98af72d800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:38:26.489 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.487+0000 7f98af72d800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:2I4xYJPrijm.f9mSMlciZFsjxofDkV9_.8pS89b1J6676Ij4Bull4CWzk-T-Ieo8_0 2023-12-03T11:38:26.490 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.488+0000 7f98917fa640 20 handle_completion(): completion ok for obj=y[2I4xYJPrijm.f9mSMlciZFsjxofDkV9] 2023-12-03T11:38:26.491 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.489+0000 7f98af72d800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:38:26.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.490+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:y state=0x561ae5061b88 s->prefetch_data=0 2023-12-03T11:38:26.492 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.490+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=3fzpatogphxvg3h95xfnnd4kyyv8zoho 2023-12-03T11:38:26.495 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.493+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.496 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.494+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.495+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.497 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.495+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:z state=0x561ae5061b58 s->prefetch_data=0 2023-12-03T11:38:26.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.495+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to 30w09fizlwo0v9vahkj8rnpc2q73pvxs 2023-12-03T11:38:26.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.495+0000 7f98af72d800 20 get_obj_state_impl: setting s->olh_tag to s1h8gjk41ciancxmtfidxf3424ec22nz 2023-12-03T11:38:26.498 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.495+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=s1h8gjk41ciancxmtfidxf3424ec22nz 2023-12-03T11:38:26.499 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.497+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.498+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.498+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=2 key=z[] 2023-12-03T11:38:26.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.498+0000 7f98af72d800 20 olh_log_entry: epoch=2 op=3 key=z[vxSrvBIAI8rVyc3IiTMW5xWyLzKnTmG] 2023-12-03T11:38:26.500 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.498+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:vxSrvBIAI8rVyc3IiTMW5xWyLzKnTmG_z state=0x561ae5067578 s->prefetch_data=0 2023-12-03T11:38:26.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.499+0000 7f98af72d800 10 manifest: total_size = 22 2023-12-03T11:38:26.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.499+0000 7f98af72d800 20 get_obj_state: setting s->obj_tag to ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8603416876339281568 2023-12-03T11:38:26.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.499+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:_:vxSrvBIAI8rVyc3IiTMW5xWyLzKnTmG_z state=0x561ae5067578 s->prefetch_data=0 2023-12-03T11:38:26.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.499+0000 7f98af72d800 20 prepare_atomic_modification: state is not atomic. state=0x561ae5067578 2023-12-03T11:38:26.501 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.499+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.501+0000 7f98af72d800 20 RGWObjManifest::operator++(): rule->part_size=0 rules.size()=1 2023-12-03T11:38:26.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.501+0000 7f98af72d800 20 RGWObjManifest::operator++(): result: ofs=22 stripe_ofs=22 part_ofs=0 rule->part_size=0 2023-12-03T11:38:26.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.501+0000 7f98af72d800 0 deleting objects inline since gc isn't initialized 2023-12-03T11:38:26.503 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.501+0000 7f98af72d800 5 delete_objs_inline: removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:vxSrvBIAI8rVyc3IiTMW5xWyLzKnTmG_.SbqdLJZFhCWbBT_UDWCH0s7V3Z0LBoq_0 2023-12-03T11:38:26.504 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.502+0000 7f9891ffb640 20 handle_completion(): completion ok for obj=z[vxSrvBIAI8rVyc3IiTMW5xWyLzKnTmG] 2023-12-03T11:38:26.505 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.502+0000 7f98af72d800 5 delete_objs_inline: refcount put returned error -2 2023-12-03T11:38:26.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.504+0000 7f98af72d800 20 get_obj_state: octx=0x7f989c06df80 obj=myver:z state=0x561ae5061b58 s->prefetch_data=0 2023-12-03T11:38:26.506 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.504+0000 7f98af72d800 20 bucket_index_guard_olh_op(): olh_state.olh_tag=s1h8gjk41ciancxmtfidxf3424ec22nz 2023-12-03T11:38:26.509 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.507+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.508+0000 7f98af72d800 20 bucket index object: default.rgw.buckets.index:.dir.ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17.3.9 2023-12-03T11:38:26.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.508+0000 7f98af72d800 1 NOTICE: finished shard 9 (3 entries removed) 2023-12-03T11:38:26.510 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.508+0000 7f98af72d800 1 NOTICE: finished all shards (3 entries removed) 2023-12-03T11:38:26.511 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.509+0000 7f98717e2640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.511+0000 7f98af72d800 20 remove_watcher() i=0 2023-12-03T11:38:26.513 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.511+0000 7f98af72d800 2 removed watcher, disabling cache 2023-12-03T11:38:26.514 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.512+0000 7f98af72d800 20 remove_watcher() i=1 2023-12-03T11:38:26.515 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.513+0000 7f98af72d800 20 remove_watcher() i=2 2023-12-03T11:38:26.516 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.514+0000 7f98af72d800 20 remove_watcher() i=3 2023-12-03T11:38:26.517 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.515+0000 7f98af72d800 20 remove_watcher() i=4 2023-12-03T11:38:26.518 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.516+0000 7f98af72d800 20 remove_watcher() i=5 2023-12-03T11:38:26.519 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.517+0000 7f98af72d800 20 remove_watcher() i=6 2023-12-03T11:38:26.520 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.518+0000 7f98af72d800 20 remove_watcher() i=7 2023-12-03T11:38:26.575 INFO:tasks.workunit.client.0.smithi012.stderr:radosgw-admin bucket stats --bucket myver 2023-12-03T11:38:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:38:26.603 INFO:tasks.workunit.client.0.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:38:26.617 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.615+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 realm 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.617+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.618+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.620 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.618+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.621 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.618+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.619+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:38:26.622 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.619+0000 7faf46d32800 20 searching for the correct realm 2023-12-03T11:38:26.631 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.629+0000 7faf43163640 0 -- 172.21.15.12:0/2389531757 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55f132128c20 msgr2=0x55f13212b0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.630+0000 7faf46d32800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.630+0000 7faf46d32800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.630+0000 7faf46d32800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.630+0000 7faf46d32800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.633 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:38:26.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:38:26.634 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 zone default found 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 4 Realm: () 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 zonegroup default 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.631+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 started sync module instance, tier type = 2023-12-03T11:38:26.635 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.632+0000 7faf46d32800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:38:26.641 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.639+0000 7faf43163640 0 -- 172.21.15.12:0/2389531757 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7faf38089920 msgr2=0x7faf38087410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:38:26.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=0 2023-12-03T11:38:26.643 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=1 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=2 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=3 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=4 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=5 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.641+0000 7faf46d32800 20 add_watcher() i=6 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.642+0000 7faf46d32800 20 add_watcher() i=7 2023-12-03T11:38:26.644 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.642+0000 7faf46d32800 2 all 8 watchers are set, enabling cache 2023-12-03T11:38:26.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.643+0000 7fae83fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.644+0000 7faf46d32800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:38:26.646 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.644+0000 7faf46d32800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:38:26.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7faf46d32800 5 note: GC not initialized 2023-12-03T11:38:26.652 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7fae72ffd640 20 reqs_thread_entry: start 2023-12-03T11:38:26.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7faf46d32800 20 init_complete bucket index max shards: 1 2023-12-03T11:38:26.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7faf46d32800 20 Filter name: none 2023-12-03T11:38:26.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7fae70ff9640 20 reqs_thread_entry: start 2023-12-03T11:38:26.653 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.650+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:26.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:26.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:26.654 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=173 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.651+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.652+0000 7faf46d32800 20 rados_obj.operate() r=0 bl.length=357 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.652+0000 7faf46d32800 20 couldn't put binfo cache entry, might have raced with data changes 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.653+0000 7faf46d32800 20 rados->read ofs=0 len=0 2023-12-03T11:38:26.655 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.653+0000 7faf46d32800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:38:26.656 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.654+0000 7fae83fff640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:38:26.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.656+0000 7faf46d32800 20 remove_watcher() i=0 2023-12-03T11:38:26.658 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.656+0000 7faf46d32800 2 removed watcher, disabling cache 2023-12-03T11:38:26.659 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.657+0000 7faf46d32800 20 remove_watcher() i=1 2023-12-03T11:38:26.660 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.658+0000 7faf46d32800 20 remove_watcher() i=2 2023-12-03T11:38:26.661 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.659+0000 7faf46d32800 20 remove_watcher() i=3 2023-12-03T11:38:26.662 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.660+0000 7faf46d32800 20 remove_watcher() i=4 2023-12-03T11:38:26.663 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.661+0000 7faf46d32800 20 remove_watcher() i=5 2023-12-03T11:38:26.664 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.662+0000 7faf46d32800 20 remove_watcher() i=6 2023-12-03T11:38:26.665 INFO:tasks.workunit.client.0.smithi012.stderr:2023-12-03T11:38:26.663+0000 7faf46d32800 20 remove_watcher() i=7 2023-12-03T11:38:26.670 INFO:tasks.workunit.client.0.smithi012.stderr:{'rgw.main': {'size': 0, 'size_actual': 0, 'size_utilized': 0, 'size_kb': 0, 'size_kb_actual': 0, 'size_kb_utilized': 0, 'num_objects': 0}} 2023-12-03T11:38:26.670 INFO:tasks.workunit.client.0.smithi012.stderr:Deleting bucket a-bucket 2023-12-03T11:38:26.687 INFO:tasks.workunit.client.0.smithi012.stderr:Deleting bucket myver 2023-12-03T11:38:26.767 INFO:tasks.workunit.client.0.smithi012.stderr:Completed resharding tests 2023-12-03T11:38:26.767 INFO:tasks.workunit.client.0.smithi012.stdout:connected to http://localhost:80 2023-12-03T11:38:26.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ deactivate 2023-12-03T11:38:26.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin ']' 2023-12-03T11:38:26.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ PATH=/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2023-12-03T11:38:26.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ export PATH 2023-12-03T11:38:26.830 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset _OLD_VIRTUAL_PATH 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n '' ']' 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n /usr/bin/bash -o -n '' ']' 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ hash -r 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' -n '' ']' 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset VIRTUAL_ENV 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ '[' '!' '' = nondestructive ']' 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ unset -f deactivate 2023-12-03T11:38:26.831 INFO:tasks.workunit.client.0.smithi012.stderr:+ echo OK. 2023-12-03T11:38:26.832 INFO:tasks.workunit.client.0.smithi012.stdout:OK. 2023-12-03T11:38:26.832 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-03T11:38:26.832 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-03T11:38:26.903 INFO:tasks.workunit:Stopping ['rgw/run-reshard.sh'] on client.0... 2023-12-03T11:38:26.903 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-03T11:38:27.302 DEBUG:teuthology.parallel:result is None 2023-12-03T11:38:27.302 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:38:27.332 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:38:27.332 DEBUG:teuthology.orchestra.run.smithi012:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-03T11:38:27.388 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-03T11:38:27.388 INFO:teuthology.run_tasks:Running task s3tests-java... 2023-12-03T11:38:27.402 DEBUG:tasks.s3tests_java:S3 Tests Java: __INIT__ 2023-12-03T11:38:27.402 DEBUG:tasks.s3tests_java:S3 Tests Java: SETUP 2023-12-03T11:38:27.402 INFO:tasks.s3tests_java:S3 Tests Java: Downloading test suite... 2023-12-03T11:38:27.403 DEBUG:teuthology.orchestra.run.smithi012:> git clone -b ceph-master https://github.com/ceph/java_s3tests.git /home/ubuntu/cephtest/s3-tests-java 2023-12-03T11:38:27.443 INFO:teuthology.orchestra.run.smithi012.stderr:Cloning into '/home/ubuntu/cephtest/s3-tests-java'... 2023-12-03T11:38:32.796 INFO:tasks.s3tests_java:S3 Tests Java: Installing required packages... 2023-12-03T11:38:32.796 DEBUG:teuthology.orchestra.run.smithi012:> /home/ubuntu/cephtest/s3-tests-java/bootstrap.sh 2023-12-03T11:38:35.928 INFO:teuthology.orchestra.run.smithi012.stderr:/home/ubuntu/cephtest/s3-tests-java/bootstrap.sh: missing required RPM packages. Installing via sudo. 2023-12-03T11:38:36.525 INFO:teuthology.orchestra.run.smithi012.stdout:Last metadata expiration check: 0:16:32 ago on Sun 03 Dec 2023 11:22:04 AM UTC. 2023-12-03T11:38:36.701 INFO:teuthology.orchestra.run.smithi012.stdout:Dependencies resolved. 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================== 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout: Package Arch Version Repository Size 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================== 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:Installing: 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 424 k 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk-devel x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 9.3 M 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout:Installing dependencies: 2023-12-03T11:38:36.709 INFO:teuthology.orchestra.run.smithi012.stdout: ModemManager-glib x86_64 1.20.2-1.el9 CentOS-BaseOS 334 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: adwaita-cursor-theme noarch 40.1.1-3.el9 CentOS-AppStream 625 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: adwaita-icon-theme noarch 40.1.1-3.el9 CentOS-AppStream 11 M 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: alsa-lib x86_64 1.2.9-1.el9 CentOS-AppStream 499 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: at-spi2-atk x86_64 2.38.0-4.el9 CentOS-AppStream 86 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: at-spi2-core x86_64 2.40.3-1.el9 CentOS-AppStream 176 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: atk x86_64 2.36.0-5.el9 CentOS-AppStream 270 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: avahi-glib x86_64 0.8-12.el9 appstream 15 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: bluez-libs x86_64 5.64-1.el9 CentOS-BaseOS 83 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: bubblewrap x86_64 0.4.1-6.el9 CentOS-BaseOS 49 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: colord-libs x86_64 1.4.5-4.el9 CentOS-AppStream 228 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: copy-jdk-configs noarch 4.0-3.el9 CentOS-AppStream 27 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: cups-libs x86_64 1:2.3.3op2-22.el9 CentOS-BaseOS 262 k 2023-12-03T11:38:36.710 INFO:teuthology.orchestra.run.smithi012.stdout: exempi x86_64 2.6.0-0.2.20211007gite23c213.el9 CentOS-AppStream 524 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: exiv2-libs x86_64 0.27.5-2.el9 CentOS-AppStream 780 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: fdk-aac-free x86_64 2.0.0-8.el9 CentOS-AppStream 324 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: flac-libs x86_64 1.3.3-12.el9 appstream 218 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak-selinux noarch 1.12.8-1.el9 CentOS-AppStream 20 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak-session-helper x86_64 1.12.8-1.el9 CentOS-AppStream 73 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gdk-pixbuf2-modules x86_64 2.42.6-3.el9 CentOS-AppStream 84 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: geoclue2 x86_64 2.6.0-7.el9 CentOS-AppStream 122 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: giflib x86_64 5.2.1-9.el9 CentOS-AppStream 47 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: graphene x86_64 1.10.6-2.el9 CentOS-AppStream 63 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gsm x86_64 1.0.19-6.el9 CentOS-AppStream 33 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gstreamer1 x86_64 1.22.1-2.el9 CentOS-AppStream 1.4 M 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gstreamer1-plugins-base x86_64 1.22.1-1.el9 CentOS-AppStream 2.2 M 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gtk-update-icon-cache x86_64 3.24.31-2.el9 CentOS-AppStream 34 k 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: gtk3 x86_64 3.24.31-2.el9 CentOS-AppStream 4.8 M 2023-12-03T11:38:36.711 INFO:teuthology.orchestra.run.smithi012.stdout: hicolor-icon-theme noarch 0.17-13.el9 CentOS-AppStream 65 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: iso-codes noarch 4.6.0-3.el9 CentOS-AppStream 3.3 M 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk-headless x86_64 1:1.8.0.362.b09-4.el9 CentOS-AppStream 32 M 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: javapackages-filesystem noarch 6.0.0-4.el9 CentOS-AppStream 10 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: jbigkit-libs x86_64 2.1-23.el9 CentOS-AppStream 52 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: lcms2 x86_64 2.12-3.el9 CentOS-AppStream 166 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libX11-xcb x86_64 1.7.0-8.el9 appstream 11 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXcomposite x86_64 0.4.5-7.el9 CentOS-AppStream 23 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXcursor x86_64 1.2.0-7.el9 CentOS-AppStream 30 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXdamage x86_64 1.1.5-7.el9 CentOS-AppStream 22 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXfixes x86_64 5.0.3-16.el9 CentOS-AppStream 19 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXi x86_64 1.7.10-8.el9 CentOS-AppStream 38 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXinerama x86_64 1.1.4-10.el9 CentOS-AppStream 14 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXrandr x86_64 1.5.2-8.el9 CentOS-AppStream 27 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXtst x86_64 1.2.3-16.el9 CentOS-AppStream 20 k 2023-12-03T11:38:36.712 INFO:teuthology.orchestra.run.smithi012.stdout: libXv x86_64 1.0.11-16.el9 CentOS-AppStream 18 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libXxf86vm x86_64 1.1.4-18.el9 CentOS-AppStream 18 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libasyncns x86_64 0.8-22.el9 CentOS-AppStream 29 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra x86_64 0.30-27.el9 CentOS-AppStream 85 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra-gtk3 x86_64 0.30-27.el9 CentOS-AppStream 31 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libdrm x86_64 2.4.117-1.el9 CentOS-AppStream 158 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libepoxy x86_64 1.5.5-4.el9 CentOS-AppStream 240 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libexif x86_64 0.6.22-6.el9 CentOS-AppStream 431 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libfontenc x86_64 1.1.3-17.el9 CentOS-AppStream 30 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libgexiv2 x86_64 0.12.3-1.el9 CentOS-AppStream 82 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd x86_64 1:1.3.4-1.el9 CentOS-AppStream 134 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd-egl x86_64 1:1.3.4-1.el9 CentOS-AppStream 36 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd-glx x86_64 1:1.3.4-1.el9 CentOS-AppStream 141 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libgsf x86_64 1.14.47-5.el9 CentOS-AppStream 245 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libgxps x86_64 0.3.2-3.el9 CentOS-AppStream 78 k 2023-12-03T11:38:36.713 INFO:teuthology.orchestra.run.smithi012.stdout: libiptcdata x86_64 1.0.5-9.el9 CentOS-AppStream 61 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libldac x86_64 2.0.2.3-10.el9 CentOS-AppStream 40 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libnotify x86_64 0.7.9-8.el9 CentOS-AppStream 43 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libogg x86_64 2:1.3.4-6.el9 CentOS-AppStream 32 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libosinfo x86_64 1.10.0-1.el9 CentOS-AppStream 312 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libpciaccess x86_64 0.16-6.el9 CentOS-BaseOS 27 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libsbc x86_64 1.4-9.el9 CentOS-AppStream 44 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libsndfile x86_64 1.0.31-8.el9 CentOS-AppStream 205 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libtheora x86_64 1:1.1.1-31.el9 CentOS-AppStream 163 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libtiff x86_64 4.4.0-10.el9 CentOS-AppStream 196 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-ltdl x86_64 2.4.6-45.el9 CentOS-AppStream 36 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libtracker-sparql x86_64 3.1.2-3.el9 CentOS-AppStream 316 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libvisual x86_64 1:0.4.0-34.el9 CentOS-AppStream 143 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libvorbis x86_64 1:1.3.7-5.el9 CentOS-AppStream 192 k 2023-12-03T11:38:36.714 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-client x86_64 1.21.0-1.el9 CentOS-AppStream 32 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-cursor x86_64 1.21.0-1.el9 CentOS-AppStream 18 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-egl x86_64 1.21.0-1.el9 CentOS-AppStream 12 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-server x86_64 1.21.0-1.el9 CentOS-AppStream 41 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libwebp x86_64 1.2.0-8.el9 CentOS-AppStream 276 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libxkbcommon x86_64 1.0.3-4.el9 CentOS-AppStream 132 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: libxshmfence x86_64 1.3-10.el9 CentOS-AppStream 12 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: lksctp-tools x86_64 1.0.19-2.el9 CentOS-BaseOS 90 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: low-memory-monitor x86_64 2.1-4.el9 CentOS-AppStream 35 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: lua-posix x86_64 35.0-8.el9 CentOS-AppStream 131 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libEGL x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 126 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libGL x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 167 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libgbm x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 36 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libglapi x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 44 k 2023-12-03T11:38:36.715 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-vulkan-drivers x86_64 23.3.0~rc2-1.el9 CentOS-AppStream 9.7 M 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: mkfontscale x86_64 1.2.1-3.el9 CentOS-AppStream 31 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: openjpeg2 x86_64 2.4.0-7.el9 CentOS-AppStream 162 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: opus x86_64 1.3.1-10.el9 CentOS-AppStream 199 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: orc x86_64 0.4.31-6.el9 CentOS-AppStream 183 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: osinfo-db noarch 20230518-1.el9 CentOS-AppStream 273 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: osinfo-db-tools x86_64 1.10.0-1.el9 CentOS-AppStream 68 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: ostree-libs x86_64 2023.7-2.el9 CentOS-AppStream 477 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-libs x86_64 0.3.67-2.el9 CentOS-AppStream 1.7 M 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: poppler x86_64 21.01.0-19.el9 CentOS-AppStream 1.1 M 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: poppler-data noarch 0.4.9-9.el9 CentOS-AppStream 1.8 M 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: poppler-glib x86_64 21.01.0-19.el9 CentOS-AppStream 151 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: pulseaudio-libs x86_64 15.0-2.el9 CentOS-AppStream 666 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: pulseaudio-utils x86_64 15.0-2.el9 CentOS-AppStream 72 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: rtkit x86_64 0.11-28.el9 CentOS-AppStream 55 k 2023-12-03T11:38:36.716 INFO:teuthology.orchestra.run.smithi012.stdout: sound-theme-freedesktop noarch 0.8-17.el9 CentOS-AppStream 377 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: totem-pl-parser x86_64 3.26.6-2.el9 CentOS-AppStream 130 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: ttmkfdir x86_64 3.0.9-65.el9 CentOS-AppStream 52 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: tzdata-java noarch 2023c-1.el9 CentOS-AppStream 149 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: upower x86_64 0.99.11-11.el9 CentOS-AppStream 168 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: vulkan-loader x86_64 1.3.250.1-1.el9 CentOS-AppStream 140 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: webrtc-audio-processing x86_64 0.3.1-8.el9 CentOS-AppStream 305 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: wireplumber x86_64 0.4.14-1.el9 CentOS-AppStream 83 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: wireplumber-libs x86_64 0.4.14-1.el9 CentOS-AppStream 337 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-dbus-proxy x86_64 0.1.3-1.el9 CentOS-AppStream 41 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-desktop-portal x86_64 1.12.4-1.el9 CentOS-AppStream 365 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: xkeyboard-config noarch 2.33-2.el9 CentOS-AppStream 780 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: xorg-x11-fonts-Type1 noarch 7.5-33.el9 CentOS-AppStream 499 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout:Installing weak dependencies: 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: dconf x86_64 0.40.0-6.el9 CentOS-AppStream 108 k 2023-12-03T11:38:36.717 INFO:teuthology.orchestra.run.smithi012.stdout: exiv2 x86_64 0.27.5-2.el9 CentOS-AppStream 978 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak x86_64 1.12.8-1.el9 CentOS-AppStream 1.7 M 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: gtk2 x86_64 2.24.33-8.el9 CentOS-AppStream 3.5 M 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra-gtk2 x86_64 0.30-27.el9 CentOS-AppStream 25 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: p11-kit-server x86_64 0.24.1-2.el9 CentOS-AppStream 198 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire x86_64 0.3.67-2.el9 CentOS-AppStream 86 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-alsa x86_64 0.3.67-2.el9 CentOS-AppStream 55 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-jack-audio-connection-kit x86_64 0.3.67-2.el9 CentOS-AppStream 126 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-pulseaudio x86_64 0.3.67-2.el9 CentOS-AppStream 18 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: tracker x86_64 3.1.2-3.el9 CentOS-AppStream 537 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: tracker-miners x86_64 3.1.2-3.el9 CentOS-AppStream 887 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-desktop-portal-gtk x86_64 1.12.0-3.el9 CentOS-AppStream 130 k 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction Summary 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:====================================================================================================== 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout:Install 129 Packages 2023-12-03T11:38:36.718 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:38:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:Total download size: 102 M 2023-12-03T11:38:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:Installed size: 381 M 2023-12-03T11:38:36.719 INFO:teuthology.orchestra.run.smithi012.stdout:Downloading Packages: 2023-12-03T11:38:36.972 INFO:teuthology.orchestra.run.smithi012.stdout:(1/129): alsa-lib-1.2.9-1.el9.x86_64.rpm 2.7 MB/s | 499 kB 00:00 2023-12-03T11:38:36.997 INFO:teuthology.orchestra.run.smithi012.stdout:(2/129): adwaita-cursor-theme-40.1.1-3.el9.noar 2.9 MB/s | 625 kB 00:00 2023-12-03T11:38:37.023 INFO:teuthology.orchestra.run.smithi012.stdout:(3/129): at-spi2-atk-2.38.0-4.el9.x86_64.rpm 1.7 MB/s | 86 kB 00:00 2023-12-03T11:38:37.048 INFO:teuthology.orchestra.run.smithi012.stdout:(4/129): at-spi2-core-2.40.3-1.el9.x86_64.rpm 3.6 MB/s | 176 kB 00:00 2023-12-03T11:38:37.082 INFO:teuthology.orchestra.run.smithi012.stdout:(5/129): atk-2.36.0-5.el9.x86_64.rpm 4.8 MB/s | 270 kB 00:00 2023-12-03T11:38:37.107 INFO:teuthology.orchestra.run.smithi012.stdout:(6/129): colord-libs-1.4.5-4.el9.x86_64.rpm 4.0 MB/s | 228 kB 00:00 2023-12-03T11:38:37.132 INFO:teuthology.orchestra.run.smithi012.stdout:(7/129): copy-jdk-configs-4.0-3.el9.noarch.rpm 578 kB/s | 27 kB 00:00 2023-12-03T11:38:37.241 INFO:teuthology.orchestra.run.smithi012.stdout:(8/129): adwaita-icon-theme-40.1.1-3.el9.noarch 24 MB/s | 11 MB 00:00 2023-12-03T11:38:37.266 INFO:teuthology.orchestra.run.smithi012.stdout:(9/129): dconf-0.40.0-6.el9.x86_64.rpm 693 kB/s | 108 kB 00:00 2023-12-03T11:38:37.291 INFO:teuthology.orchestra.run.smithi012.stdout:(10/129): exempi-2.6.0-0.2.20211007gite23c213.e 3.2 MB/s | 524 kB 00:00 2023-12-03T11:38:37.325 INFO:teuthology.orchestra.run.smithi012.stdout:(11/129): exiv2-0.27.5-2.el9.x86_64.rpm 17 MB/s | 978 kB 00:00 2023-12-03T11:38:37.358 INFO:teuthology.orchestra.run.smithi012.stdout:(12/129): exiv2-libs-0.27.5-2.el9.x86_64.rpm 8.4 MB/s | 780 kB 00:00 2023-12-03T11:38:37.384 INFO:teuthology.orchestra.run.smithi012.stdout:(13/129): fdk-aac-free-2.0.0-8.el9.x86_64.rpm 3.5 MB/s | 324 kB 00:00 2023-12-03T11:38:37.425 INFO:teuthology.orchestra.run.smithi012.stdout:(14/129): flatpak-1.12.8-1.el9.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-12-03T11:38:37.451 INFO:teuthology.orchestra.run.smithi012.stdout:(15/129): flatpak-selinux-1.12.8-1.el9.noarch.r 314 kB/s | 20 kB 00:00 2023-12-03T11:38:37.476 INFO:teuthology.orchestra.run.smithi012.stdout:(16/129): flatpak-session-helper-1.12.8-1.el9.x 814 kB/s | 73 kB 00:00 2023-12-03T11:38:37.501 INFO:teuthology.orchestra.run.smithi012.stdout:(17/129): gdk-pixbuf2-modules-2.42.6-3.el9.x86_ 1.1 MB/s | 84 kB 00:00 2023-12-03T11:38:37.526 INFO:teuthology.orchestra.run.smithi012.stdout:(18/129): geoclue2-2.6.0-7.el9.x86_64.rpm 2.4 MB/s | 122 kB 00:00 2023-12-03T11:38:37.552 INFO:teuthology.orchestra.run.smithi012.stdout:(19/129): giflib-5.2.1-9.el9.x86_64.rpm 630 kB/s | 47 kB 00:00 2023-12-03T11:38:37.577 INFO:teuthology.orchestra.run.smithi012.stdout:(20/129): graphene-1.10.6-2.el9.x86_64.rpm 841 kB/s | 63 kB 00:00 2023-12-03T11:38:37.602 INFO:teuthology.orchestra.run.smithi012.stdout:(21/129): gsm-1.0.19-6.el9.x86_64.rpm 661 kB/s | 33 kB 00:00 2023-12-03T11:38:37.636 INFO:teuthology.orchestra.run.smithi012.stdout:(22/129): gstreamer1-1.22.1-2.el9.x86_64.rpm 17 MB/s | 1.4 MB 00:00 2023-12-03T11:38:37.686 INFO:teuthology.orchestra.run.smithi012.stdout:(23/129): gstreamer1-plugins-base-1.22.1-1.el9. 20 MB/s | 2.2 MB 00:00 2023-12-03T11:38:37.711 INFO:teuthology.orchestra.run.smithi012.stdout:(24/129): gtk-update-icon-cache-3.24.31-2.el9.x 468 kB/s | 34 kB 00:00 2023-12-03T11:38:37.770 INFO:teuthology.orchestra.run.smithi012.stdout:(25/129): gtk2-2.24.33-8.el9.x86_64.rpm 26 MB/s | 3.5 MB 00:00 2023-12-03T11:38:37.879 INFO:teuthology.orchestra.run.smithi012.stdout:(26/129): gtk3-3.24.31-2.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-12-03T11:38:37.904 INFO:teuthology.orchestra.run.smithi012.stdout:(27/129): hicolor-icon-theme-0.17-13.el9.noarch 345 kB/s | 65 kB 00:00 2023-12-03T11:38:37.979 INFO:teuthology.orchestra.run.smithi012.stdout:(28/129): iso-codes-4.6.0-3.el9.noarch.rpm 16 MB/s | 3.3 MB 00:00 2023-12-03T11:38:38.005 INFO:teuthology.orchestra.run.smithi012.stdout:(29/129): java-1.8.0-openjdk-1.8.0.362.b09-4.el 3.4 MB/s | 424 kB 00:00 2023-12-03T11:38:38.188 INFO:teuthology.orchestra.run.smithi012.stdout:(30/129): java-1.8.0-openjdk-devel-1.8.0.362.b0 33 MB/s | 9.3 MB 00:00 2023-12-03T11:38:38.205 INFO:teuthology.orchestra.run.smithi012.stdout:(31/129): javapackages-filesystem-6.0.0-4.el9.n 53 kB/s | 10 kB 00:00 2023-12-03T11:38:38.397 INFO:teuthology.orchestra.run.smithi012.stdout:(32/129): jbigkit-libs-2.1-23.el9.x86_64.rpm 250 kB/s | 52 kB 00:00 2023-12-03T11:38:38.422 INFO:teuthology.orchestra.run.smithi012.stdout:(33/129): lcms2-2.12-3.el9.x86_64.rpm 777 kB/s | 166 kB 00:00 2023-12-03T11:38:38.589 INFO:teuthology.orchestra.run.smithi012.stdout:(34/129): libXcomposite-0.4.5-7.el9.x86_64.rpm 122 kB/s | 23 kB 00:00 2023-12-03T11:38:38.614 INFO:teuthology.orchestra.run.smithi012.stdout:(35/129): libXcursor-1.2.0-7.el9.x86_64.rpm 156 kB/s | 30 kB 00:00 2023-12-03T11:38:38.915 INFO:teuthology.orchestra.run.smithi012.stdout:(36/129): java-1.8.0-openjdk-headless-1.8.0.362 36 MB/s | 32 MB 00:00 2023-12-03T11:38:38.940 INFO:teuthology.orchestra.run.smithi012.stdout:(37/129): libXdamage-1.1.5-7.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2023-12-03T11:38:38.965 INFO:teuthology.orchestra.run.smithi012.stdout:(38/129): libXfixes-5.0.3-16.el9.x86_64.rpm 53 kB/s | 19 kB 00:00 2023-12-03T11:38:38.990 INFO:teuthology.orchestra.run.smithi012.stdout:(39/129): libXi-1.7.10-8.el9.x86_64.rpm 515 kB/s | 38 kB 00:00 2023-12-03T11:38:39.016 INFO:teuthology.orchestra.run.smithi012.stdout:(40/129): libXrandr-1.5.2-8.el9.x86_64.rpm 550 kB/s | 27 kB 00:00 2023-12-03T11:38:39.041 INFO:teuthology.orchestra.run.smithi012.stdout:(41/129): libXinerama-1.1.4-10.el9.x86_64.rpm 186 kB/s | 14 kB 00:00 2023-12-03T11:38:39.066 INFO:teuthology.orchestra.run.smithi012.stdout:(42/129): libXtst-1.2.3-16.el9.x86_64.rpm 271 kB/s | 20 kB 00:00 2023-12-03T11:38:39.091 INFO:teuthology.orchestra.run.smithi012.stdout:(43/129): libXv-1.0.11-16.el9.x86_64.rpm 239 kB/s | 18 kB 00:00 2023-12-03T11:38:39.117 INFO:teuthology.orchestra.run.smithi012.stdout:(44/129): libXxf86vm-1.1.4-18.el9.x86_64.rpm 355 kB/s | 18 kB 00:00 2023-12-03T11:38:39.142 INFO:teuthology.orchestra.run.smithi012.stdout:(45/129): libasyncns-0.8-22.el9.x86_64.rpm 392 kB/s | 29 kB 00:00 2023-12-03T11:38:39.167 INFO:teuthology.orchestra.run.smithi012.stdout:(46/129): libcanberra-0.30-27.el9.x86_64.rpm 1.1 MB/s | 85 kB 00:00 2023-12-03T11:38:39.192 INFO:teuthology.orchestra.run.smithi012.stdout:(47/129): libcanberra-gtk2-0.30-27.el9.x86_64.r 499 kB/s | 25 kB 00:00 2023-12-03T11:38:39.218 INFO:teuthology.orchestra.run.smithi012.stdout:(48/129): libcanberra-gtk3-0.30-27.el9.x86_64.r 410 kB/s | 31 kB 00:00 2023-12-03T11:38:39.243 INFO:teuthology.orchestra.run.smithi012.stdout:(49/129): libdrm-2.4.117-1.el9.x86_64.rpm 2.1 MB/s | 158 kB 00:00 2023-12-03T11:38:39.268 INFO:teuthology.orchestra.run.smithi012.stdout:(50/129): libepoxy-1.5.5-4.el9.x86_64.rpm 4.7 MB/s | 240 kB 00:00 2023-12-03T11:38:39.293 INFO:teuthology.orchestra.run.smithi012.stdout:(51/129): libexif-0.6.22-6.el9.x86_64.rpm 5.6 MB/s | 431 kB 00:00 2023-12-03T11:38:39.319 INFO:teuthology.orchestra.run.smithi012.stdout:(52/129): libfontenc-1.1.3-17.el9.x86_64.rpm 403 kB/s | 30 kB 00:00 2023-12-03T11:38:39.344 INFO:teuthology.orchestra.run.smithi012.stdout:(53/129): libgexiv2-0.12.3-1.el9.x86_64.rpm 1.6 MB/s | 82 kB 00:00 2023-12-03T11:38:39.369 INFO:teuthology.orchestra.run.smithi012.stdout:(54/129): libglvnd-1.3.4-1.el9.x86_64.rpm 1.7 MB/s | 134 kB 00:00 2023-12-03T11:38:39.394 INFO:teuthology.orchestra.run.smithi012.stdout:(55/129): libglvnd-egl-1.3.4-1.el9.x86_64.rpm 476 kB/s | 36 kB 00:00 2023-12-03T11:38:39.420 INFO:teuthology.orchestra.run.smithi012.stdout:(56/129): libglvnd-glx-1.3.4-1.el9.x86_64.rpm 2.8 MB/s | 141 kB 00:00 2023-12-03T11:38:39.445 INFO:teuthology.orchestra.run.smithi012.stdout:(57/129): libgsf-1.14.47-5.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2023-12-03T11:38:39.470 INFO:teuthology.orchestra.run.smithi012.stdout:(58/129): libgxps-0.3.2-3.el9.x86_64.rpm 1.0 MB/s | 78 kB 00:00 2023-12-03T11:38:39.495 INFO:teuthology.orchestra.run.smithi012.stdout:(59/129): libiptcdata-1.0.5-9.el9.x86_64.rpm 1.2 MB/s | 61 kB 00:00 2023-12-03T11:38:39.521 INFO:teuthology.orchestra.run.smithi012.stdout:(60/129): libldac-2.0.2.3-10.el9.x86_64.rpm 536 kB/s | 40 kB 00:00 2023-12-03T11:38:39.546 INFO:teuthology.orchestra.run.smithi012.stdout:(61/129): libnotify-0.7.9-8.el9.x86_64.rpm 567 kB/s | 43 kB 00:00 2023-12-03T11:38:39.571 INFO:teuthology.orchestra.run.smithi012.stdout:(62/129): libogg-1.3.4-6.el9.x86_64.rpm 653 kB/s | 32 kB 00:00 2023-12-03T11:38:39.596 INFO:teuthology.orchestra.run.smithi012.stdout:(63/129): libosinfo-1.10.0-1.el9.x86_64.rpm 4.1 MB/s | 312 kB 00:00 2023-12-03T11:38:39.622 INFO:teuthology.orchestra.run.smithi012.stdout:(64/129): libsbc-1.4-9.el9.x86_64.rpm 585 kB/s | 44 kB 00:00 2023-12-03T11:38:39.647 INFO:teuthology.orchestra.run.smithi012.stdout:(65/129): libsndfile-1.0.31-8.el9.x86_64.rpm 4.0 MB/s | 205 kB 00:00 2023-12-03T11:38:39.672 INFO:teuthology.orchestra.run.smithi012.stdout:(66/129): libtheora-1.1.1-31.el9.x86_64.rpm 2.1 MB/s | 163 kB 00:00 2023-12-03T11:38:39.698 INFO:teuthology.orchestra.run.smithi012.stdout:(67/129): libtiff-4.4.0-10.el9.x86_64.rpm 2.6 MB/s | 196 kB 00:00 2023-12-03T11:38:39.723 INFO:teuthology.orchestra.run.smithi012.stdout:(68/129): libtool-ltdl-2.4.6-45.el9.x86_64.rpm 726 kB/s | 36 kB 00:00 2023-12-03T11:38:39.748 INFO:teuthology.orchestra.run.smithi012.stdout:(69/129): libtracker-sparql-3.1.2-3.el9.x86_64. 4.1 MB/s | 316 kB 00:00 2023-12-03T11:38:39.773 INFO:teuthology.orchestra.run.smithi012.stdout:(70/129): libvisual-0.4.0-34.el9.x86_64.rpm 1.9 MB/s | 143 kB 00:00 2023-12-03T11:38:39.798 INFO:teuthology.orchestra.run.smithi012.stdout:(71/129): libvorbis-1.3.7-5.el9.x86_64.rpm 3.8 MB/s | 192 kB 00:00 2023-12-03T11:38:39.824 INFO:teuthology.orchestra.run.smithi012.stdout:(72/129): libwayland-client-1.21.0-1.el9.x86_64 432 kB/s | 32 kB 00:00 2023-12-03T11:38:39.849 INFO:teuthology.orchestra.run.smithi012.stdout:(73/129): libwayland-cursor-1.21.0-1.el9.x86_64 245 kB/s | 18 kB 00:00 2023-12-03T11:38:39.874 INFO:teuthology.orchestra.run.smithi012.stdout:(74/129): libwayland-egl-1.21.0-1.el9.x86_64.rp 235 kB/s | 12 kB 00:00 2023-12-03T11:38:39.899 INFO:teuthology.orchestra.run.smithi012.stdout:(75/129): libwayland-server-1.21.0-1.el9.x86_64 541 kB/s | 41 kB 00:00 2023-12-03T11:38:39.925 INFO:teuthology.orchestra.run.smithi012.stdout:(76/129): libwebp-1.2.0-8.el9.x86_64.rpm 3.6 MB/s | 276 kB 00:00 2023-12-03T11:38:39.950 INFO:teuthology.orchestra.run.smithi012.stdout:(77/129): libxshmfence-1.3-10.el9.x86_64.rpm 234 kB/s | 12 kB 00:00 2023-12-03T11:38:39.967 INFO:teuthology.orchestra.run.smithi012.stdout:(78/129): libxkbcommon-1.0.3-4.el9.x86_64.rpm 1.9 MB/s | 132 kB 00:00 2023-12-03T11:38:39.992 INFO:teuthology.orchestra.run.smithi012.stdout:(79/129): low-memory-monitor-2.1-4.el9.x86_64.r 522 kB/s | 35 kB 00:00 2023-12-03T11:38:40.017 INFO:teuthology.orchestra.run.smithi012.stdout:(80/129): lua-posix-35.0-8.el9.x86_64.rpm 1.9 MB/s | 131 kB 00:00 2023-12-03T11:38:40.043 INFO:teuthology.orchestra.run.smithi012.stdout:(81/129): mesa-libEGL-23.3.0~rc2-1.el9.x86_64.r 2.5 MB/s | 126 kB 00:00 2023-12-03T11:38:40.068 INFO:teuthology.orchestra.run.smithi012.stdout:(82/129): mesa-libGL-23.3.0~rc2-1.el9.x86_64.rp 2.2 MB/s | 167 kB 00:00 2023-12-03T11:38:40.093 INFO:teuthology.orchestra.run.smithi012.stdout:(83/129): mesa-libgbm-23.3.0~rc2-1.el9.x86_64.r 486 kB/s | 36 kB 00:00 2023-12-03T11:38:40.118 INFO:teuthology.orchestra.run.smithi012.stdout:(84/129): mesa-libglapi-23.3.0~rc2-1.el9.x86_64 882 kB/s | 44 kB 00:00 2023-12-03T11:38:40.327 INFO:teuthology.orchestra.run.smithi012.stdout:(85/129): mesa-vulkan-drivers-23.3.0~rc2-1.el9. 38 MB/s | 9.7 MB 00:00 2023-12-03T11:38:40.352 INFO:teuthology.orchestra.run.smithi012.stdout:(86/129): mkfontscale-1.2.1-3.el9.x86_64.rpm 121 kB/s | 31 kB 00:00 2023-12-03T11:38:40.377 INFO:teuthology.orchestra.run.smithi012.stdout:(87/129): openjpeg2-2.4.0-7.el9.x86_64.rpm 634 kB/s | 162 kB 00:00 2023-12-03T11:38:40.411 INFO:teuthology.orchestra.run.smithi012.stdout:(88/129): opus-1.3.1-10.el9.x86_64.rpm 2.3 MB/s | 199 kB 00:00 2023-12-03T11:38:40.436 INFO:teuthology.orchestra.run.smithi012.stdout:(89/129): orc-0.4.31-6.el9.x86_64.rpm 3.1 MB/s | 183 kB 00:00 2023-12-03T11:38:40.462 INFO:teuthology.orchestra.run.smithi012.stdout:(90/129): osinfo-db-20230518-1.el9.noarch.rpm 3.2 MB/s | 273 kB 00:00 2023-12-03T11:38:40.487 INFO:teuthology.orchestra.run.smithi012.stdout:(91/129): osinfo-db-tools-1.10.0-1.el9.x86_64.r 912 kB/s | 68 kB 00:00 2023-12-03T11:38:40.512 INFO:teuthology.orchestra.run.smithi012.stdout:(92/129): p11-kit-server-0.24.1-2.el9.x86_64.rp 3.9 MB/s | 198 kB 00:00 2023-12-03T11:38:40.546 INFO:teuthology.orchestra.run.smithi012.stdout:(93/129): ostree-libs-2023.7-2.el9.x86_64.rpm 5.6 MB/s | 477 kB 00:00 2023-12-03T11:38:40.571 INFO:teuthology.orchestra.run.smithi012.stdout:(94/129): pipewire-0.3.67-2.el9.x86_64.rpm 1.0 MB/s | 86 kB 00:00 2023-12-03T11:38:40.596 INFO:teuthology.orchestra.run.smithi012.stdout:(95/129): pipewire-alsa-0.3.67-2.el9.x86_64.rpm 656 kB/s | 55 kB 00:00 2023-12-03T11:38:40.622 INFO:teuthology.orchestra.run.smithi012.stdout:(96/129): pipewire-jack-audio-connection-kit-0. 2.5 MB/s | 126 kB 00:00 2023-12-03T11:38:40.664 INFO:teuthology.orchestra.run.smithi012.stdout:(97/129): pipewire-libs-0.3.67-2.el9.x86_64.rpm 19 MB/s | 1.7 MB 00:00 2023-12-03T11:38:40.689 INFO:teuthology.orchestra.run.smithi012.stdout:(98/129): pipewire-pulseaudio-0.3.67-2.el9.x86_ 197 kB/s | 18 kB 00:00 2023-12-03T11:38:40.731 INFO:teuthology.orchestra.run.smithi012.stdout:(99/129): poppler-21.01.0-19.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2023-12-03T11:38:40.756 INFO:teuthology.orchestra.run.smithi012.stdout:(100/129): poppler-glib-21.01.0-19.el9.x86_64.r 2.3 MB/s | 151 kB 00:00 2023-12-03T11:38:40.798 INFO:teuthology.orchestra.run.smithi012.stdout:(101/129): poppler-data-0.4.9-9.el9.noarch.rpm 17 MB/s | 1.8 MB 00:00 2023-12-03T11:38:40.823 INFO:teuthology.orchestra.run.smithi012.stdout:(102/129): pulseaudio-libs-15.0-2.el9.x86_64.rp 7.1 MB/s | 666 kB 00:00 2023-12-03T11:38:40.857 INFO:teuthology.orchestra.run.smithi012.stdout:(103/129): pulseaudio-utils-15.0-2.el9.x86_64.r 735 kB/s | 72 kB 00:00 2023-12-03T11:38:40.882 INFO:teuthology.orchestra.run.smithi012.stdout:(104/129): rtkit-0.11-28.el9.x86_64.rpm 943 kB/s | 55 kB 00:00 2023-12-03T11:38:40.907 INFO:teuthology.orchestra.run.smithi012.stdout:(105/129): sound-theme-freedesktop-0.8-17.el9.n 4.4 MB/s | 377 kB 00:00 2023-12-03T11:38:40.932 INFO:teuthology.orchestra.run.smithi012.stdout:(106/129): totem-pl-parser-3.26.6-2.el9.x86_64. 1.7 MB/s | 130 kB 00:00 2023-12-03T11:38:40.974 INFO:teuthology.orchestra.run.smithi012.stdout:(107/129): tracker-3.1.2-3.el9.x86_64.rpm 7.9 MB/s | 537 kB 00:00 2023-12-03T11:38:41.008 INFO:teuthology.orchestra.run.smithi012.stdout:(108/129): tracker-miners-3.1.2-3.el9.x86_64.rp 8.7 MB/s | 887 kB 00:00 2023-12-03T11:38:41.025 INFO:teuthology.orchestra.run.smithi012.stdout:(109/129): ttmkfdir-3.0.9-65.el9.x86_64.rpm 568 kB/s | 52 kB 00:00 2023-12-03T11:38:41.058 INFO:teuthology.orchestra.run.smithi012.stdout:(110/129): tzdata-java-2023c-1.el9.noarch.rpm 1.7 MB/s | 149 kB 00:00 2023-12-03T11:38:41.084 INFO:teuthology.orchestra.run.smithi012.stdout:(111/129): upower-0.99.11-11.el9.x86_64.rpm 2.8 MB/s | 168 kB 00:00 2023-12-03T11:38:41.109 INFO:teuthology.orchestra.run.smithi012.stdout:(112/129): vulkan-loader-1.3.250.1-1.el9.x86_64 1.6 MB/s | 140 kB 00:00 2023-12-03T11:38:41.134 INFO:teuthology.orchestra.run.smithi012.stdout:(113/129): webrtc-audio-processing-0.3.1-8.el9. 4.0 MB/s | 305 kB 00:00 2023-12-03T11:38:41.159 INFO:teuthology.orchestra.run.smithi012.stdout:(114/129): wireplumber-0.4.14-1.el9.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-12-03T11:38:41.185 INFO:teuthology.orchestra.run.smithi012.stdout:(115/129): wireplumber-libs-0.4.14-1.el9.x86_64 4.4 MB/s | 337 kB 00:00 2023-12-03T11:38:41.210 INFO:teuthology.orchestra.run.smithi012.stdout:(116/129): xdg-dbus-proxy-0.1.3-1.el9.x86_64.rp 554 kB/s | 41 kB 00:00 2023-12-03T11:38:41.235 INFO:teuthology.orchestra.run.smithi012.stdout:(117/129): xdg-desktop-portal-1.12.4-1.el9.x86_ 7.1 MB/s | 365 kB 00:00 2023-12-03T11:38:41.260 INFO:teuthology.orchestra.run.smithi012.stdout:(118/129): xdg-desktop-portal-gtk-1.12.0-3.el9. 1.7 MB/s | 130 kB 00:00 2023-12-03T11:38:41.294 INFO:teuthology.orchestra.run.smithi012.stdout:(119/129): xkeyboard-config-2.33-2.el9.noarch.r 9.2 MB/s | 780 kB 00:00 2023-12-03T11:38:41.328 INFO:teuthology.orchestra.run.smithi012.stdout:(120/129): xorg-x11-fonts-Type1-7.5-33.el9.noar 7.5 MB/s | 499 kB 00:00 2023-12-03T11:38:41.353 INFO:teuthology.orchestra.run.smithi012.stdout:(121/129): ModemManager-glib-1.20.2-1.el9.x86_6 3.6 MB/s | 334 kB 00:00 2023-12-03T11:38:41.378 INFO:teuthology.orchestra.run.smithi012.stdout:(122/129): bluez-libs-5.64-1.el9.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-12-03T11:38:41.404 INFO:teuthology.orchestra.run.smithi012.stdout:(123/129): bubblewrap-0.4.1-6.el9.x86_64.rpm 980 kB/s | 49 kB 00:00 2023-12-03T11:38:41.464 INFO:teuthology.orchestra.run.smithi012.stdout:(124/129): cups-libs-2.3.3op2-22.el9.x86_64.rpm 2.3 MB/s | 262 kB 00:00 2023-12-03T11:38:41.488 INFO:teuthology.orchestra.run.smithi012.stdout:(125/129): libpciaccess-0.16-6.el9.x86_64.rpm 245 kB/s | 27 kB 00:00 2023-12-03T11:38:41.513 INFO:teuthology.orchestra.run.smithi012.stdout:(126/129): lksctp-tools-1.0.19-2.el9.x86_64.rpm 1.8 MB/s | 90 kB 00:00 2023-12-03T11:38:41.822 INFO:teuthology.orchestra.run.smithi012.stdout:(127/129): avahi-glib-0.8-12.el9.x86_64.rpm 41 kB/s | 15 kB 00:00 2023-12-03T11:38:41.872 INFO:teuthology.orchestra.run.smithi012.stdout:(128/129): libX11-xcb-1.7.0-8.el9.x86_64.rpm 30 kB/s | 11 kB 00:00 2023-12-03T11:38:42.147 INFO:teuthology.orchestra.run.smithi012.stdout:(129/129): flac-libs-1.3.3-12.el9.x86_64.rpm 331 kB/s | 218 kB 00:00 2023-12-03T11:38:42.153 INFO:teuthology.orchestra.run.smithi012.stdout:-------------------------------------------------------------------------------- 2023-12-03T11:38:42.153 INFO:teuthology.orchestra.run.smithi012.stdout:Total 19 MB/s | 102 MB 00:05 2023-12-03T11:38:42.189 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction check 2023-12-03T11:38:42.294 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction check succeeded. 2023-12-03T11:38:42.295 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction test 2023-12-03T11:38:43.364 INFO:teuthology.orchestra.run.smithi012.stdout:Transaction test succeeded. 2023-12-03T11:38:43.364 INFO:teuthology.orchestra.run.smithi012.stdout:Running transaction 2023-12-03T11:38:47.757 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: copy-jdk-configs-4.0-3.el9.noarch 1/1 2023-12-03T11:38:47.823 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el9.x8 1/1 2023-12-03T11:38:47.984 INFO:teuthology.orchestra.run.smithi012.stdout: Preparing : 1/1 2023-12-03T11:38:48.058 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXi-1.7.10-8.el9.x86_64 1/129 2023-12-03T11:38:48.093 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libwayland-client-1.21.0-1.el9.x86_64 2/129 2023-12-03T11:38:48.123 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libogg-2:1.3.4-6.el9.x86_64 3/129 2023-12-03T11:38:48.152 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libvorbis-1:1.3.7-5.el9.x86_64 4/129 2023-12-03T11:38:48.279 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXfixes-5.0.3-16.el9.x86_64 5/129 2023-12-03T11:38:48.359 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : atk-2.36.0-5.el9.x86_64 6/129 2023-12-03T11:38:48.390 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : alsa-lib-1.2.9-1.el9.x86_64 7/129 2023-12-03T11:38:48.429 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libX11-xcb-1.7.0-8.el9.x86_64 8/129 2023-12-03T11:38:48.460 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : cups-libs-1:2.3.3op2-22.el9.x86_64 9/129 2023-12-03T11:38:48.483 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : opus-1.3.1-10.el9.x86_64 10/129 2023-12-03T11:38:48.509 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libxshmfence-1.3-10.el9.x86_64 11/129 2023-12-03T11:38:48.536 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtool-ltdl-2.4.6-45.el9.x86_64 12/129 2023-12-03T11:38:48.563 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXcomposite-0.4.5-7.el9.x86_64 13/129 2023-12-03T11:38:48.656 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lcms2-2.12-3.el9.x86_64 14/129 2023-12-03T11:38:48.700 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gstreamer1-1.22.1-2.el9.x86_64 15/129 2023-12-03T11:38:48.739 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXcursor-1.2.0-7.el9.x86_64 16/129 2023-12-03T11:38:48.765 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXdamage-1.1.5-7.el9.x86_64 17/129 2023-12-03T11:38:48.791 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libwayland-cursor-1.21.0-1.el9.x86_64 18/129 2023-12-03T11:38:48.817 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXtst-1.2.3-16.el9.x86_64 19/129 2023-12-03T11:38:48.842 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mesa-libglapi-23.3.0~rc2-1.el9.x86_64 20/129 2023-12-03T11:38:48.867 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libwayland-server-1.21.0-1.el9.x86_64 21/129 2023-12-03T11:38:48.897 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libwayland-egl-1.21.0-1.el9.x86_64 22/129 2023-12-03T11:38:48.927 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libglvnd-1:1.3.4-1.el9.x86_64 23/129 2023-12-03T11:38:48.952 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXrandr-1.5.2-8.el9.x86_64 24/129 2023-12-03T11:38:49.077 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXinerama-1.1.4-10.el9.x86_64 25/129 2023-12-03T11:38:49.134 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : hicolor-icon-theme-0.17-13.el9.noarch 26/129 2023-12-03T11:38:49.189 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gtk-update-icon-cache-3.24.31-2.el9.x86_64 27/129 2023-12-03T11:38:49.219 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : dconf-0.40.0-6.el9.x86_64 28/129 2023-12-03T11:38:49.284 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 28/129 2023-12-03T11:38:49.321 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : at-spi2-core-2.40.3-1.el9.x86_64 29/129 2023-12-03T11:38:49.357 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : at-spi2-atk-2.38.0-4.el9.x86_64 30/129 2023-12-03T11:38:49.388 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : colord-libs-1.4.5-4.el9.x86_64 31/129 2023-12-03T11:38:49.424 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtheora-1:1.1.1-31.el9.x86_64 32/129 2023-12-03T11:38:49.465 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flac-libs-1.3.3-12.el9.x86_64 33/129 2023-12-03T11:38:49.524 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : exiv2-libs-0.27.5-2.el9.x86_64 34/129 2023-12-03T11:38:49.561 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : exiv2-0.27.5-2.el9.x86_64 35/129 2023-12-03T11:38:49.588 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgexiv2-0.12.3-1.el9.x86_64 36/129 2023-12-03T11:38:49.621 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : avahi-glib-0.8-12.el9.x86_64 37/129 2023-12-03T11:38:49.649 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 38/129 2023-12-03T11:38:49.680 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libpciaccess-0.16-6.el9.x86_64 39/129 2023-12-03T11:38:49.707 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libdrm-2.4.117-1.el9.x86_64 40/129 2023-12-03T11:38:49.734 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mesa-libgbm-23.3.0~rc2-1.el9.x86_64 41/129 2023-12-03T11:38:49.760 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libglvnd-egl-1:1.3.4-1.el9.x86_64 42/129 2023-12-03T11:38:49.789 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mesa-libEGL-23.3.0~rc2-1.el9.x86_64 43/129 2023-12-03T11:38:50.026 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : vulkan-loader-1.3.250.1-1.el9.x86_64 44/129 2023-12-03T11:38:50.063 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 45/129 2023-12-03T11:38:50.087 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bubblewrap-0.4.1-6.el9.x86_64 46/129 2023-12-03T11:38:50.120 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : bluez-libs-5.64-1.el9.x86_64 47/129 2023-12-03T11:38:50.329 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ModemManager-glib-1.20.2-1.el9.x86_64 48/129 2023-12-03T11:38:50.377 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xkeyboard-config-2.33-2.el9.noarch 49/129 2023-12-03T11:38:50.409 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libxkbcommon-1.0.3-4.el9.x86_64 50/129 2023-12-03T11:38:50.439 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xdg-dbus-proxy-0.1.3-1.el9.x86_64 51/129 2023-12-03T11:38:50.480 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : webrtc-audio-processing-0.3.1-8.el9.x86_64 52/129 2023-12-03T11:38:50.508 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : upower-0.99.11-11.el9.x86_64 53/129 2023-12-03T11:38:50.522 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: upower-0.99.11-11.el9.x86_64 53/129 2023-12-03T11:38:50.522 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/graphical.target.wants/upower.service → /usr/lib/systemd/system/upower.service. 2023-12-03T11:38:50.522 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:38:50.721 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : tzdata-java-2023c-1.el9.noarch 54/129 2023-12-03T11:38:50.791 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ttmkfdir-3.0.9-65.el9.x86_64 55/129 2023-12-03T11:38:50.842 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : totem-pl-parser-3.26.6-2.el9.x86_64 56/129 2023-12-03T11:38:50.866 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : sound-theme-freedesktop-0.8-17.el9.noarch 57/129 2023-12-03T11:38:50.872 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: sound-theme-freedesktop-0.8-17.el9.noarch 57/129 2023-12-03T11:38:51.222 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:38:51.245 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:38:51.258 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: rtkit-0.11-28.el9.x86_64 58/129 2023-12-03T11:38:51.258 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/graphical.target.wants/rtkit-daemon.service → /usr/lib/systemd/system/rtkit-daemon.service. 2023-12-03T11:38:51.258 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:38:51.479 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : poppler-data-0.4.9-9.el9.noarch 59/129 2023-12-03T11:38:51.513 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : p11-kit-server-0.24.1-2.el9.x86_64 60/129 2023-12-03T11:38:51.553 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : ostree-libs-2023.7-2.el9.x86_64 61/129 2023-12-03T11:38:52.058 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : osinfo-db-tools-1.10.0-1.el9.x86_64 62/129 2023-12-03T11:38:52.115 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : osinfo-db-20230518-1.el9.noarch 63/129 2023-12-03T11:38:52.153 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libosinfo-1.10.0-1.el9.x86_64 64/129 2023-12-03T11:38:52.187 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : orc-0.4.31-6.el9.x86_64 65/129 2023-12-03T11:38:52.242 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : openjpeg2-2.4.0-7.el9.x86_64 66/129 2023-12-03T11:38:52.277 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : lua-posix-35.0-8.el9.x86_64 67/129 2023-12-03T11:38:52.305 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : copy-jdk-configs-4.0-3.el9.noarch 68/129 2023-12-03T11:38:52.334 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : low-memory-monitor-2.1-4.el9.x86_64 69/129 2023-12-03T11:38:52.346 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: low-memory-monitor-2.1-4.el9.x86_64 69/129 2023-12-03T11:38:52.346 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/basic.target.wants/low-memory-monitor.service → /usr/lib/systemd/system/low-memory-monitor.service. 2023-12-03T11:38:52.346 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:38:52.393 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libwebp-1.2.0-8.el9.x86_64 70/129 2023-12-03T11:38:52.421 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libvisual-1:0.4.0-34.el9.x86_64 71/129 2023-12-03T11:38:52.449 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libsbc-1.4-9.el9.x86_64 72/129 2023-12-03T11:38:52.481 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libnotify-0.7.9-8.el9.x86_64 73/129 2023-12-03T11:38:52.843 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:38:52.894 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:38:52.913 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: geoclue2-2.6.0-7.el9.x86_64 74/129 2023-12-03T11:38:52.943 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libldac-2.0.2.3-10.el9.x86_64 75/129 2023-12-03T11:38:53.001 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libiptcdata-1.0.5-9.el9.x86_64 76/129 2023-12-03T11:38:53.035 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgsf-1.14.47-5.el9.x86_64 77/129 2023-12-03T11:38:53.060 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libfontenc-1.1.3-17.el9.x86_64 78/129 2023-12-03T11:38:53.103 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mkfontscale-1.2.1-3.el9.x86_64 79/129 2023-12-03T11:38:53.141 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xorg-x11-fonts-Type1-7.5-33.el9.noarch 80/129 2023-12-03T11:38:53.406 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: xorg-x11-fonts-Type1-7.5-33.el9.noarch 80/129 2023-12-03T11:38:53.443 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libexif-0.6.22-6.el9.x86_64 81/129 2023-12-03T11:38:53.471 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libepoxy-1.5.5-4.el9.x86_64 82/129 2023-12-03T11:38:53.497 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libasyncns-0.8-22.el9.x86_64 83/129 2023-12-03T11:38:53.525 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXxf86vm-1.1.4-18.el9.x86_64 84/129 2023-12-03T11:38:53.557 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libglvnd-glx-1:1.3.4-1.el9.x86_64 85/129 2023-12-03T11:38:53.581 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : mesa-libGL-23.3.0~rc2-1.el9.x86_64 86/129 2023-12-03T11:38:53.609 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libXv-1.0.11-16.el9.x86_64 87/129 2023-12-03T11:38:53.645 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : jbigkit-libs-2.1-23.el9.x86_64 88/129 2023-12-03T11:38:53.691 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtiff-4.4.0-10.el9.x86_64 89/129 2023-12-03T11:38:53.721 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 90/129 2023-12-03T11:38:53.770 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libgxps-0.3.2-3.el9.x86_64 91/129 2023-12-03T11:38:53.804 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : poppler-21.01.0-19.el9.x86_64 92/129 2023-12-03T11:38:53.838 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : poppler-glib-21.01.0-19.el9.x86_64 93/129 2023-12-03T11:38:54.463 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : javapackages-filesystem-6.0.0-4.el9.noarch 94/129 2023-12-03T11:38:54.506 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 95/129 2023-12-03T11:38:55.198 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 95/129 2023-12-03T11:38:55.250 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : iso-codes-4.6.0-3.el9.noarch 96/129 2023-12-03T11:38:55.278 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gsm-1.0.19-6.el9.x86_64 97/129 2023-12-03T11:38:55.348 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libsndfile-1.0.31-8.el9.x86_64 98/129 2023-12-03T11:38:55.385 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pulseaudio-libs-15.0-2.el9.x86_64 99/129 2023-12-03T11:38:55.410 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcanberra-0.30-27.el9.x86_64 100/129 2023-12-03T11:38:55.435 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: libcanberra-0.30-27.el9.x86_64 100/129 2023-12-03T11:38:55.481 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pulseaudio-utils-15.0-2.el9.x86_64 101/129 2023-12-03T11:38:55.603 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : graphene-1.10.6-2.el9.x86_64 102/129 2023-12-03T11:38:55.645 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gstreamer1-plugins-base-1.22.1-1.el9.x86_64 103/129 2023-12-03T11:38:55.679 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : giflib-5.2.1-9.el9.x86_64 104/129 2023-12-03T11:38:55.710 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flatpak-session-helper-1.12.8-1.el9.x86_64 105/129 2023-12-03T11:38:55.734 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flatpak-selinux-1.12.8-1.el9.noarch 106/129 2023-12-03T11:39:06.135 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: flatpak-selinux-1.12.8-1.el9.noarch 106/129 2023-12-03T11:39:06.165 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : fdk-aac-free-2.0.0-8.el9.x86_64 107/129 2023-12-03T11:39:06.637 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:39:06.661 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:39:06.677 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: pipewire-0.3.67-2.el9.x86_64 108/129 2023-12-03T11:39:06.677 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire.socket → /usr/lib/systemd/user/pipewire.socket. 2023-12-03T11:39:06.677 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:06.897 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pipewire-libs-0.3.67-2.el9.x86_64 109/129 2023-12-03T11:39:06.967 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : wireplumber-0.4.14-1.el9.x86_64 110/129 2023-12-03T11:39:06.997 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : wireplumber-libs-0.4.14-1.el9.x86_64 111/129 2023-12-03T11:39:07.510 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: flatpak-1.12.8-1.el9.x86_64 112/129 2023-12-03T11:39:07.595 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : flatpak-1.12.8-1.el9.x86_64 112/129 2023-12-03T11:39:07.632 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xdg-desktop-portal-1.12.4-1.el9.x86_64 113/129 2023-12-03T11:39:07.664 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: xdg-desktop-portal-1.12.4-1.el9.x86_64 113/129 2023-12-03T11:39:07.716 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 114/129 2023-12-03T11:39:07.784 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libtracker-sparql-3.1.2-3.el9.x86_64 115/129 2023-12-03T11:39:07.813 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : tracker-3.1.2-3.el9.x86_64 116/129 2023-12-03T11:39:07.926 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: tracker-3.1.2-3.el9.x86_64 116/129 2023-12-03T11:39:07.968 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : tracker-miners-3.1.2-3.el9.x86_64 117/129 2023-12-03T11:39:08.061 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: tracker-miners-3.1.2-3.el9.x86_64 117/129 2023-12-03T11:39:11.072 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : adwaita-cursor-theme-40.1.1-3.el9.noarch 118/129 2023-12-03T11:39:11.162 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : adwaita-icon-theme-40.1.1-3.el9.noarch 119/129 2023-12-03T11:39:11.206 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcanberra-gtk3-0.30-27.el9.x86_64 120/129 2023-12-03T11:39:11.233 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 121/129 2023-12-03T11:39:11.428 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 121/129 2023-12-03T11:39:11.581 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gtk3-3.24.31-2.el9.x86_64 122/129 2023-12-03T11:39:11.621 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : gtk2-2.24.33-8.el9.x86_64 123/129 2023-12-03T11:39:11.673 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : libcanberra-gtk2-0.30-27.el9.x86_64 124/129 2023-12-03T11:39:11.704 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 125/129 2023-12-03T11:39:11.965 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 125/129 2023-12-03T11:39:12.000 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 126/129 2023-12-03T11:39:12.011 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 126/129 2023-12-03T11:39:12.057 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pipewire-alsa-0.3.67-2.el9.x86_64 127/129 2023-12-03T11:39:12.087 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pipewire-jack-audio-connection-kit-0.3.67-2.el9. 128/129 2023-12-03T11:39:12.110 INFO:teuthology.orchestra.run.smithi012.stdout: Installing : pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:39:12.125 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:39:12.125 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/user/sockets.target.wants/pipewire-pulse.socket → /usr/lib/systemd/user/pipewire-pulse.socket. 2023-12-03T11:39:12.125 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:12.150 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: dconf-0.40.0-6.el9.x86_64 129/129 2023-12-03T11:39:12.156 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: copy-jdk-configs-4.0-3.el9.noarch 129/129 2023-12-03T11:39:12.181 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 129/129 2023-12-03T11:39:12.194 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: wireplumber-0.4.14-1.el9.x86_64 129/129 2023-12-03T11:39:12.194 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/user/pipewire-session-manager.service → /usr/lib/systemd/user/wireplumber.service. 2023-12-03T11:39:12.194 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/user/pipewire.service.wants/wireplumber.service → /usr/lib/systemd/user/wireplumber.service. 2023-12-03T11:39:12.194 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:12.257 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 129/129 2023-12-03T11:39:12.333 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 129/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Running scriptlet: pipewire-pulseaudio-0.3.67-2.el9.x86_64 129/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : adwaita-cursor-theme-40.1.1-3.el9.noarch 1/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : adwaita-icon-theme-40.1.1-3.el9.noarch 2/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : alsa-lib-1.2.9-1.el9.x86_64 3/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : at-spi2-atk-2.38.0-4.el9.x86_64 4/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : at-spi2-core-2.40.3-1.el9.x86_64 5/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : atk-2.36.0-5.el9.x86_64 6/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : colord-libs-1.4.5-4.el9.x86_64 7/129 2023-12-03T11:39:15.942 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : copy-jdk-configs-4.0-3.el9.noarch 8/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : dconf-0.40.0-6.el9.x86_64 9/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 10/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : exiv2-0.27.5-2.el9.x86_64 11/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : exiv2-libs-0.27.5-2.el9.x86_64 12/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : fdk-aac-free-2.0.0-8.el9.x86_64 13/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flatpak-1.12.8-1.el9.x86_64 14/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flatpak-selinux-1.12.8-1.el9.noarch 15/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flatpak-session-helper-1.12.8-1.el9.x86_64 16/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 17/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : geoclue2-2.6.0-7.el9.x86_64 18/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : giflib-5.2.1-9.el9.x86_64 19/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : graphene-1.10.6-2.el9.x86_64 20/129 2023-12-03T11:39:15.943 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gsm-1.0.19-6.el9.x86_64 21/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gstreamer1-1.22.1-2.el9.x86_64 22/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gstreamer1-plugins-base-1.22.1-1.el9.x86_64 23/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gtk-update-icon-cache-3.24.31-2.el9.x86_64 24/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gtk2-2.24.33-8.el9.x86_64 25/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : gtk3-3.24.31-2.el9.x86_64 26/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : hicolor-icon-theme-0.17-13.el9.noarch 27/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : iso-codes-4.6.0-3.el9.noarch 28/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 29/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x 30/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el 31/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : javapackages-filesystem-6.0.0-4.el9.noarch 32/129 2023-12-03T11:39:15.944 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : jbigkit-libs-2.1-23.el9.x86_64 33/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lcms2-2.12-3.el9.x86_64 34/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXcomposite-0.4.5-7.el9.x86_64 35/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXcursor-1.2.0-7.el9.x86_64 36/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXdamage-1.1.5-7.el9.x86_64 37/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXfixes-5.0.3-16.el9.x86_64 38/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXi-1.7.10-8.el9.x86_64 39/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXinerama-1.1.4-10.el9.x86_64 40/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXrandr-1.5.2-8.el9.x86_64 41/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXtst-1.2.3-16.el9.x86_64 42/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXv-1.0.11-16.el9.x86_64 43/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libXxf86vm-1.1.4-18.el9.x86_64 44/129 2023-12-03T11:39:15.945 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libasyncns-0.8-22.el9.x86_64 45/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcanberra-0.30-27.el9.x86_64 46/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcanberra-gtk2-0.30-27.el9.x86_64 47/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libcanberra-gtk3-0.30-27.el9.x86_64 48/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libdrm-2.4.117-1.el9.x86_64 49/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libepoxy-1.5.5-4.el9.x86_64 50/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libexif-0.6.22-6.el9.x86_64 51/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libfontenc-1.1.3-17.el9.x86_64 52/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgexiv2-0.12.3-1.el9.x86_64 53/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libglvnd-1:1.3.4-1.el9.x86_64 54/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libglvnd-egl-1:1.3.4-1.el9.x86_64 55/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libglvnd-glx-1:1.3.4-1.el9.x86_64 56/129 2023-12-03T11:39:15.946 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgsf-1.14.47-5.el9.x86_64 57/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libgxps-0.3.2-3.el9.x86_64 58/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libiptcdata-1.0.5-9.el9.x86_64 59/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libldac-2.0.2.3-10.el9.x86_64 60/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libnotify-0.7.9-8.el9.x86_64 61/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libogg-2:1.3.4-6.el9.x86_64 62/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libosinfo-1.10.0-1.el9.x86_64 63/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsbc-1.4-9.el9.x86_64 64/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libsndfile-1.0.31-8.el9.x86_64 65/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtheora-1:1.1.1-31.el9.x86_64 66/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtiff-4.4.0-10.el9.x86_64 67/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtool-ltdl-2.4.6-45.el9.x86_64 68/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libtracker-sparql-3.1.2-3.el9.x86_64 69/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libvisual-1:0.4.0-34.el9.x86_64 70/129 2023-12-03T11:39:15.947 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libvorbis-1:1.3.7-5.el9.x86_64 71/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libwayland-client-1.21.0-1.el9.x86_64 72/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libwayland-cursor-1.21.0-1.el9.x86_64 73/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libwayland-egl-1.21.0-1.el9.x86_64 74/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libwayland-server-1.21.0-1.el9.x86_64 75/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libwebp-1.2.0-8.el9.x86_64 76/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libxkbcommon-1.0.3-4.el9.x86_64 77/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libxshmfence-1.3-10.el9.x86_64 78/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : low-memory-monitor-2.1-4.el9.x86_64 79/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lua-posix-35.0-8.el9.x86_64 80/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mesa-libEGL-23.3.0~rc2-1.el9.x86_64 81/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mesa-libGL-23.3.0~rc2-1.el9.x86_64 82/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mesa-libgbm-23.3.0~rc2-1.el9.x86_64 83/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mesa-libglapi-23.3.0~rc2-1.el9.x86_64 84/129 2023-12-03T11:39:15.948 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 85/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : mkfontscale-1.2.1-3.el9.x86_64 86/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : openjpeg2-2.4.0-7.el9.x86_64 87/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : opus-1.3.1-10.el9.x86_64 88/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : orc-0.4.31-6.el9.x86_64 89/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : osinfo-db-20230518-1.el9.noarch 90/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : osinfo-db-tools-1.10.0-1.el9.x86_64 91/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ostree-libs-2023.7-2.el9.x86_64 92/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : p11-kit-server-0.24.1-2.el9.x86_64 93/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pipewire-0.3.67-2.el9.x86_64 94/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pipewire-alsa-0.3.67-2.el9.x86_64 95/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pipewire-jack-audio-connection-kit-0.3.67-2.el9. 96/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pipewire-libs-0.3.67-2.el9.x86_64 97/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pipewire-pulseaudio-0.3.67-2.el9.x86_64 98/129 2023-12-03T11:39:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : poppler-21.01.0-19.el9.x86_64 99/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : poppler-data-0.4.9-9.el9.noarch 100/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : poppler-glib-21.01.0-19.el9.x86_64 101/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pulseaudio-libs-15.0-2.el9.x86_64 102/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : pulseaudio-utils-15.0-2.el9.x86_64 103/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : rtkit-0.11-28.el9.x86_64 104/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : sound-theme-freedesktop-0.8-17.el9.noarch 105/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : totem-pl-parser-3.26.6-2.el9.x86_64 106/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : tracker-3.1.2-3.el9.x86_64 107/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : tracker-miners-3.1.2-3.el9.x86_64 108/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ttmkfdir-3.0.9-65.el9.x86_64 109/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : tzdata-java-2023c-1.el9.noarch 110/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : upower-0.99.11-11.el9.x86_64 111/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : vulkan-loader-1.3.250.1-1.el9.x86_64 112/129 2023-12-03T11:39:15.950 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : webrtc-audio-processing-0.3.1-8.el9.x86_64 113/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : wireplumber-0.4.14-1.el9.x86_64 114/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : wireplumber-libs-0.4.14-1.el9.x86_64 115/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xdg-dbus-proxy-0.1.3-1.el9.x86_64 116/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xdg-desktop-portal-1.12.4-1.el9.x86_64 117/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 118/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xkeyboard-config-2.33-2.el9.noarch 119/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : xorg-x11-fonts-Type1-7.5-33.el9.noarch 120/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : ModemManager-glib-1.20.2-1.el9.x86_64 121/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bluez-libs-5.64-1.el9.x86_64 122/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : bubblewrap-0.4.1-6.el9.x86_64 123/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : cups-libs-1:2.3.3op2-22.el9.x86_64 124/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libpciaccess-0.16-6.el9.x86_64 125/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 126/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : avahi-glib-0.8-12.el9.x86_64 127/129 2023-12-03T11:39:15.951 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : flac-libs-1.3.3-12.el9.x86_64 128/129 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: Verifying : libX11-xcb-1.7.0-8.el9.x86_64 129/129 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout:Installed: 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: ModemManager-glib-1.20.2-1.el9.x86_64 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: adwaita-cursor-theme-40.1.1-3.el9.noarch 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: adwaita-icon-theme-40.1.1-3.el9.noarch 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: alsa-lib-1.2.9-1.el9.x86_64 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: at-spi2-atk-2.38.0-4.el9.x86_64 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: at-spi2-core-2.40.3-1.el9.x86_64 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: atk-2.36.0-5.el9.x86_64 2023-12-03T11:39:17.897 INFO:teuthology.orchestra.run.smithi012.stdout: avahi-glib-0.8-12.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: bluez-libs-5.64-1.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: bubblewrap-0.4.1-6.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: colord-libs-1.4.5-4.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: copy-jdk-configs-4.0-3.el9.noarch 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: cups-libs-1:2.3.3op2-22.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: dconf-0.40.0-6.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: exempi-2.6.0-0.2.20211007gite23c213.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: exiv2-0.27.5-2.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: exiv2-libs-0.27.5-2.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: fdk-aac-free-2.0.0-8.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: flac-libs-1.3.3-12.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak-1.12.8-1.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak-selinux-1.12.8-1.el9.noarch 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: flatpak-session-helper-1.12.8-1.el9.x86_64 2023-12-03T11:39:17.898 INFO:teuthology.orchestra.run.smithi012.stdout: gdk-pixbuf2-modules-2.42.6-3.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: geoclue2-2.6.0-7.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: giflib-5.2.1-9.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: graphene-1.10.6-2.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gsm-1.0.19-6.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gstreamer1-1.22.1-2.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gstreamer1-plugins-base-1.22.1-1.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gtk-update-icon-cache-3.24.31-2.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gtk2-2.24.33-8.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: gtk3-3.24.31-2.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: hicolor-icon-theme-0.17-13.el9.noarch 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: iso-codes-4.6.0-3.el9.noarch 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk-devel-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:39:17.899 INFO:teuthology.orchestra.run.smithi012.stdout: java-1.8.0-openjdk-headless-1:1.8.0.362.b09-4.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: javapackages-filesystem-6.0.0-4.el9.noarch 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: jbigkit-libs-2.1-23.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: lcms2-2.12-3.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libX11-xcb-1.7.0-8.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXcomposite-0.4.5-7.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXcursor-1.2.0-7.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXdamage-1.1.5-7.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXfixes-5.0.3-16.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXi-1.7.10-8.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXinerama-1.1.4-10.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXrandr-1.5.2-8.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXtst-1.2.3-16.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXv-1.0.11-16.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libXxf86vm-1.1.4-18.el9.x86_64 2023-12-03T11:39:17.900 INFO:teuthology.orchestra.run.smithi012.stdout: libasyncns-0.8-22.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra-0.30-27.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra-gtk2-0.30-27.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libcanberra-gtk3-0.30-27.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libdrm-2.4.117-1.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libepoxy-1.5.5-4.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libexif-0.6.22-6.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libfontenc-1.1.3-17.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libgexiv2-0.12.3-1.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd-1:1.3.4-1.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd-egl-1:1.3.4-1.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libglvnd-glx-1:1.3.4-1.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libgsf-1.14.47-5.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libgxps-0.3.2-3.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libiptcdata-1.0.5-9.el9.x86_64 2023-12-03T11:39:17.901 INFO:teuthology.orchestra.run.smithi012.stdout: libldac-2.0.2.3-10.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libnotify-0.7.9-8.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libogg-2:1.3.4-6.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libosinfo-1.10.0-1.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libpciaccess-0.16-6.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libsbc-1.4-9.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libsndfile-1.0.31-8.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libtheora-1:1.1.1-31.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libtiff-4.4.0-10.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libtool-ltdl-2.4.6-45.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libtracker-sparql-3.1.2-3.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libvisual-1:0.4.0-34.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libvorbis-1:1.3.7-5.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-client-1.21.0-1.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-cursor-1.21.0-1.el9.x86_64 2023-12-03T11:39:17.902 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-egl-1.21.0-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: libwayland-server-1.21.0-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: libwebp-1.2.0-8.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: libxkbcommon-1.0.3-4.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: libxshmfence-1.3-10.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: low-memory-monitor-2.1-4.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: lua-posix-35.0-8.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libEGL-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libGL-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libgbm-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-libglapi-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mesa-vulkan-drivers-23.3.0~rc2-1.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: mkfontscale-1.2.1-3.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: openjpeg2-2.4.0-7.el9.x86_64 2023-12-03T11:39:17.903 INFO:teuthology.orchestra.run.smithi012.stdout: opus-1.3.1-10.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: orc-0.4.31-6.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: osinfo-db-20230518-1.el9.noarch 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: osinfo-db-tools-1.10.0-1.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: ostree-libs-2023.7-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: p11-kit-server-0.24.1-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-0.3.67-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-alsa-0.3.67-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-jack-audio-connection-kit-0.3.67-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-libs-0.3.67-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pipewire-pulseaudio-0.3.67-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: poppler-21.01.0-19.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: poppler-data-0.4.9-9.el9.noarch 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: poppler-glib-21.01.0-19.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pulseaudio-libs-15.0-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: pulseaudio-utils-15.0-2.el9.x86_64 2023-12-03T11:39:17.904 INFO:teuthology.orchestra.run.smithi012.stdout: rtkit-0.11-28.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: sound-theme-freedesktop-0.8-17.el9.noarch 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: totem-pl-parser-3.26.6-2.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: tracker-3.1.2-3.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: tracker-miners-3.1.2-3.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: ttmkfdir-3.0.9-65.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: tzdata-java-2023c-1.el9.noarch 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: upower-0.99.11-11.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: vulkan-loader-1.3.250.1-1.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: webrtc-audio-processing-0.3.1-8.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: wireplumber-0.4.14-1.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: wireplumber-libs-0.4.14-1.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-dbus-proxy-0.1.3-1.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-desktop-portal-1.12.4-1.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: xdg-desktop-portal-gtk-1.12.0-3.el9.x86_64 2023-12-03T11:39:17.905 INFO:teuthology.orchestra.run.smithi012.stdout: xkeyboard-config-2.33-2.el9.noarch 2023-12-03T11:39:17.906 INFO:teuthology.orchestra.run.smithi012.stdout: xorg-x11-fonts-Type1-7.5-33.el9.noarch 2023-12-03T11:39:17.906 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:17.906 INFO:teuthology.orchestra.run.smithi012.stdout:Complete! 2023-12-03T11:39:18.199 INFO:teuthology.orchestra.run.smithi012.stdout: Gradle will be installed under /opt/gradle 2023-12-03T11:39:18.377 INFO:teuthology.orchestra.run.smithi012.stderr:--2023-12-03 11:39:18-- https://services.gradle.org/distributions/gradle-6.0.1-bin.zip 2023-12-03T11:39:18.536 INFO:teuthology.orchestra.run.smithi012.stderr:Resolving services.gradle.org (services.gradle.org)... 104.17.128.37, 104.17.129.37, 2606:4700::6811:8125, ... 2023-12-03T11:39:18.574 INFO:teuthology.orchestra.run.smithi012.stderr:Connecting to services.gradle.org (services.gradle.org)|104.17.128.37|:443... connected. 2023-12-03T11:39:18.734 INFO:teuthology.orchestra.run.smithi012.stderr:HTTP request sent, awaiting response... 301 Moved Permanently 2023-12-03T11:39:18.734 INFO:teuthology.orchestra.run.smithi012.stderr:Location: https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip [following] 2023-12-03T11:39:18.734 INFO:teuthology.orchestra.run.smithi012.stderr:--2023-12-03 11:39:18-- https://github.com/gradle/gradle-distributions/releases/download/v6.0.1/gradle-6.0.1-bin.zip 2023-12-03T11:39:18.735 INFO:teuthology.orchestra.run.smithi012.stderr:Resolving github.com (github.com)... 140.82.113.4 2023-12-03T11:39:18.792 INFO:teuthology.orchestra.run.smithi012.stderr:Connecting to github.com (github.com)|140.82.113.4|:443... connected. 2023-12-03T11:39:19.003 INFO:teuthology.orchestra.run.smithi012.stderr:HTTP request sent, awaiting response... 302 Found 2023-12-03T11:39:19.003 INFO:teuthology.orchestra.run.smithi012.stderr:Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20231203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231203T113918Z&X-Amz-Expires=300&X-Amz-Signature=d7b00910412b51137f2f5c54a025a1b07f2977299e9eb9e0bddb177c26367e71&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream [following] 2023-12-03T11:39:19.004 INFO:teuthology.orchestra.run.smithi012.stderr:--2023-12-03 11:39:19-- https://objects.githubusercontent.com/github-production-release-asset-2e65be/696192900/8f418ba1-c336-448c-95c6-5180382113cb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20231203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20231203T113918Z&X-Amz-Expires=300&X-Amz-Signature=d7b00910412b51137f2f5c54a025a1b07f2977299e9eb9e0bddb177c26367e71&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=696192900&response-content-disposition=attachment%3B%20filename%3Dgradle-6.0.1-bin.zip&response-content-type=application%2Foctet-stream 2023-12-03T11:39:19.004 INFO:teuthology.orchestra.run.smithi012.stderr:Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.111.133, 185.199.109.133, 185.199.110.133, ... 2023-12-03T11:39:19.052 INFO:teuthology.orchestra.run.smithi012.stderr:Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.111.133|:443... connected. 2023-12-03T11:39:19.146 INFO:teuthology.orchestra.run.smithi012.stderr:HTTP request sent, awaiting response... 200 OK 2023-12-03T11:39:19.146 INFO:teuthology.orchestra.run.smithi012.stderr:Length: 95129672 (91M) [application/octet-stream] 2023-12-03T11:39:19.146 INFO:teuthology.orchestra.run.smithi012.stderr:Saving to: ‘gradle-6.0.1-bin.zip’ 2023-12-03T11:39:19.146 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:39:19.182 INFO:teuthology.orchestra.run.smithi012.stderr: 0K .......... .......... .......... .......... .......... 0% 1.33M 68s 2023-12-03T11:39:19.200 INFO:teuthology.orchestra.run.smithi012.stderr: 50K .......... .......... .......... .......... .......... 0% 2.67M 51s 2023-12-03T11:39:19.214 INFO:teuthology.orchestra.run.smithi012.stderr: 100K .......... .......... .......... .......... .......... 0% 3.56M 43s 2023-12-03T11:39:19.223 INFO:teuthology.orchestra.run.smithi012.stderr: 150K .......... .......... .......... .......... .......... 0% 5.62M 36s 2023-12-03T11:39:19.244 INFO:teuthology.orchestra.run.smithi012.stderr: 200K .......... .......... .......... .......... .......... 0% 2.30M 37s 2023-12-03T11:39:19.269 INFO:teuthology.orchestra.run.smithi012.stderr: 250K .......... .......... .......... .......... .......... 0% 1.89M 38s 2023-12-03T11:39:19.282 INFO:teuthology.orchestra.run.smithi012.stderr: 300K .......... .......... .......... .......... .......... 0% 3.82M 36s 2023-12-03T11:39:19.343 INFO:teuthology.orchestra.run.smithi012.stderr: 350K .......... .......... .......... .......... .......... 0% 857K 45s 2023-12-03T11:39:19.343 INFO:teuthology.orchestra.run.smithi012.stderr: 400K .......... .......... .......... .......... .......... 0% 249M 40s 2023-12-03T11:39:19.343 INFO:teuthology.orchestra.run.smithi012.stderr: 450K .......... .......... .......... .......... .......... 0% 358M 36s 2023-12-03T11:39:19.353 INFO:teuthology.orchestra.run.smithi012.stderr: 500K .......... .......... .......... .......... .......... 0% 4.19M 35s 2023-12-03T11:39:19.367 INFO:teuthology.orchestra.run.smithi012.stderr: 550K .......... .......... .......... .......... .......... 0% 3.37M 34s 2023-12-03T11:39:19.384 INFO:teuthology.orchestra.run.smithi012.stderr: 600K .......... .......... .......... .......... .......... 0% 2.95M 34s 2023-12-03T11:39:19.404 INFO:teuthology.orchestra.run.smithi012.stderr: 650K .......... .......... .......... .......... .......... 0% 2.35M 34s 2023-12-03T11:39:19.422 INFO:teuthology.orchestra.run.smithi012.stderr: 700K .......... .......... .......... .......... .......... 0% 2.84M 34s 2023-12-03T11:39:19.463 INFO:teuthology.orchestra.run.smithi012.stderr: 750K .......... .......... .......... .......... .......... 0% 1.84M 35s 2023-12-03T11:39:19.476 INFO:teuthology.orchestra.run.smithi012.stderr: 800K .......... .......... .......... .......... .......... 0% 1.78M 36s 2023-12-03T11:39:19.503 INFO:teuthology.orchestra.run.smithi012.stderr: 850K .......... .......... .......... .......... .......... 0% 1.83M 37s 2023-12-03T11:39:19.529 INFO:teuthology.orchestra.run.smithi012.stderr: 900K .......... .......... .......... .......... .......... 1% 1.87M 37s 2023-12-03T11:39:19.553 INFO:teuthology.orchestra.run.smithi012.stderr: 950K .......... .......... .......... .......... .......... 1% 1.99M 37s 2023-12-03T11:39:19.625 INFO:teuthology.orchestra.run.smithi012.stderr: 1000K .......... .......... .......... .......... .......... 1% 698K 42s 2023-12-03T11:39:19.633 INFO:teuthology.orchestra.run.smithi012.stderr: 1050K .......... .......... .......... .......... .......... 1% 5.97M 41s 2023-12-03T11:39:19.657 INFO:teuthology.orchestra.run.smithi012.stderr: 1100K .......... .......... .......... .......... .......... 1% 2.16M 41s 2023-12-03T11:39:19.657 INFO:teuthology.orchestra.run.smithi012.stderr: 1150K .......... .......... .......... .......... .......... 1% 210M 39s 2023-12-03T11:39:19.674 INFO:teuthology.orchestra.run.smithi012.stderr: 1200K .......... .......... .......... .......... .......... 1% 2.72M 39s 2023-12-03T11:39:19.705 INFO:teuthology.orchestra.run.smithi012.stderr: 1250K .......... .......... .......... .......... .......... 1% 1.55M 39s 2023-12-03T11:39:19.730 INFO:teuthology.orchestra.run.smithi012.stderr: 1300K .......... .......... .......... .......... .......... 1% 1.99M 40s 2023-12-03T11:39:19.758 INFO:teuthology.orchestra.run.smithi012.stderr: 1350K .......... .......... .......... .......... .......... 1% 1.72M 40s 2023-12-03T11:39:19.793 INFO:teuthology.orchestra.run.smithi012.stderr: 1400K .......... .......... .......... .......... .......... 1% 1.42M 41s 2023-12-03T11:39:19.828 INFO:teuthology.orchestra.run.smithi012.stderr: 1450K .......... .......... .......... .......... .......... 1% 1.39M 42s 2023-12-03T11:39:19.862 INFO:teuthology.orchestra.run.smithi012.stderr: 1500K .......... .......... .......... .......... .......... 1% 1.42M 42s 2023-12-03T11:39:19.907 INFO:teuthology.orchestra.run.smithi012.stderr: 1550K .......... .......... .......... .......... .......... 1% 1.38M 43s 2023-12-03T11:39:19.939 INFO:teuthology.orchestra.run.smithi012.stderr: 1600K .......... .......... .......... .......... .......... 1% 1.18M 44s 2023-12-03T11:39:19.964 INFO:teuthology.orchestra.run.smithi012.stderr: 1650K .......... .......... .......... .......... .......... 1% 1.92M 44s 2023-12-03T11:39:19.983 INFO:teuthology.orchestra.run.smithi012.stderr: 1700K .......... .......... .......... .......... .......... 1% 2.66M 44s 2023-12-03T11:39:20.003 INFO:teuthology.orchestra.run.smithi012.stderr: 1750K .......... .......... .......... .......... .......... 1% 2.39M 43s 2023-12-03T11:39:20.021 INFO:teuthology.orchestra.run.smithi012.stderr: 1800K .......... .......... .......... .......... .......... 1% 2.73M 43s 2023-12-03T11:39:20.041 INFO:teuthology.orchestra.run.smithi012.stderr: 1850K .......... .......... .......... .......... .......... 2% 2.47M 43s 2023-12-03T11:39:20.061 INFO:teuthology.orchestra.run.smithi012.stderr: 1900K .......... .......... .......... .......... .......... 2% 2.47M 43s 2023-12-03T11:39:20.086 INFO:teuthology.orchestra.run.smithi012.stderr: 1950K .......... .......... .......... .......... .......... 2% 2.40M 43s 2023-12-03T11:39:20.106 INFO:teuthology.orchestra.run.smithi012.stderr: 2000K .......... .......... .......... .......... .......... 2% 1.94M 43s 2023-12-03T11:39:20.129 INFO:teuthology.orchestra.run.smithi012.stderr: 2050K .......... .......... .......... .......... .......... 2% 2.14M 43s 2023-12-03T11:39:20.163 INFO:teuthology.orchestra.run.smithi012.stderr: 2100K .......... .......... .......... .......... .......... 2% 1.44M 43s 2023-12-03T11:39:20.202 INFO:teuthology.orchestra.run.smithi012.stderr: 2150K .......... .......... .......... .......... .......... 2% 1.23M 44s 2023-12-03T11:39:20.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2200K .......... .......... .......... .......... .......... 2% 1.26M 44s 2023-12-03T11:39:20.265 INFO:teuthology.orchestra.run.smithi012.stderr: 2250K .......... .......... .......... .......... .......... 2% 2.00M 44s 2023-12-03T11:39:20.291 INFO:teuthology.orchestra.run.smithi012.stderr: 2300K .......... .......... .......... .......... .......... 2% 1.89M 44s 2023-12-03T11:39:20.324 INFO:teuthology.orchestra.run.smithi012.stderr: 2350K .......... .......... .......... .......... .......... 2% 1.91M 44s 2023-12-03T11:39:20.344 INFO:teuthology.orchestra.run.smithi012.stderr: 2400K .......... .......... .......... .......... .......... 2% 1.79M 44s 2023-12-03T11:39:20.362 INFO:teuthology.orchestra.run.smithi012.stderr: 2450K .......... .......... .......... .......... .......... 2% 2.79M 44s 2023-12-03T11:39:20.382 INFO:teuthology.orchestra.run.smithi012.stderr: 2500K .......... .......... .......... .......... .......... 2% 2.41M 44s 2023-12-03T11:39:20.403 INFO:teuthology.orchestra.run.smithi012.stderr: 2550K .......... .......... .......... .......... .......... 2% 2.30M 44s 2023-12-03T11:39:20.421 INFO:teuthology.orchestra.run.smithi012.stderr: 2600K .......... .......... .......... .......... .......... 2% 2.69M 43s 2023-12-03T11:39:20.441 INFO:teuthology.orchestra.run.smithi012.stderr: 2650K .......... .......... .......... .......... .......... 2% 2.48M 43s 2023-12-03T11:39:20.459 INFO:teuthology.orchestra.run.smithi012.stderr: 2700K .......... .......... .......... .......... .......... 2% 2.68M 43s 2023-12-03T11:39:20.483 INFO:teuthology.orchestra.run.smithi012.stderr: 2750K .......... .......... .......... .......... .......... 3% 2.58M 43s 2023-12-03T11:39:20.504 INFO:teuthology.orchestra.run.smithi012.stderr: 2800K .......... .......... .......... .......... .......... 3% 1.90M 43s 2023-12-03T11:39:20.519 INFO:teuthology.orchestra.run.smithi012.stderr: 2850K .......... .......... .......... .......... .......... 3% 3.22M 43s 2023-12-03T11:39:20.537 INFO:teuthology.orchestra.run.smithi012.stderr: 2900K .......... .......... .......... .......... .......... 3% 2.65M 42s 2023-12-03T11:39:20.554 INFO:teuthology.orchestra.run.smithi012.stderr: 2950K .......... .......... .......... .......... .......... 3% 2.88M 42s 2023-12-03T11:39:20.572 INFO:teuthology.orchestra.run.smithi012.stderr: 3000K .......... .......... .......... .......... .......... 3% 2.67M 42s 2023-12-03T11:39:20.591 INFO:teuthology.orchestra.run.smithi012.stderr: 3050K .......... .......... .......... .......... .......... 3% 2.58M 42s 2023-12-03T11:39:20.614 INFO:teuthology.orchestra.run.smithi012.stderr: 3100K .......... .......... .......... .......... .......... 3% 2.19M 42s 2023-12-03T11:39:20.644 INFO:teuthology.orchestra.run.smithi012.stderr: 3150K .......... .......... .......... .......... .......... 3% 2.01M 42s 2023-12-03T11:39:20.667 INFO:teuthology.orchestra.run.smithi012.stderr: 3200K .......... .......... .......... .......... .......... 3% 1.69M 42s 2023-12-03T11:39:20.690 INFO:teuthology.orchestra.run.smithi012.stderr: 3250K .......... .......... .......... .......... .......... 3% 2.12M 42s 2023-12-03T11:39:20.719 INFO:teuthology.orchestra.run.smithi012.stderr: 3300K .......... .......... .......... .......... .......... 3% 1.66M 42s 2023-12-03T11:39:20.745 INFO:teuthology.orchestra.run.smithi012.stderr: 3350K .......... .......... .......... .......... .......... 3% 1.96M 42s 2023-12-03T11:39:20.766 INFO:teuthology.orchestra.run.smithi012.stderr: 3400K .......... .......... .......... .......... .......... 3% 2.25M 42s 2023-12-03T11:39:20.792 INFO:teuthology.orchestra.run.smithi012.stderr: 3450K .......... .......... .......... .......... .......... 3% 1.87M 42s 2023-12-03T11:39:20.800 INFO:teuthology.orchestra.run.smithi012.stderr: 3500K .......... .......... .......... .......... .......... 3% 6.18M 42s 2023-12-03T11:39:20.820 INFO:teuthology.orchestra.run.smithi012.stderr: 3550K .......... .......... .......... .......... .......... 3% 3.17M 41s 2023-12-03T11:39:20.831 INFO:teuthology.orchestra.run.smithi012.stderr: 3600K .......... .......... .......... .......... .......... 3% 3.13M 41s 2023-12-03T11:39:20.842 INFO:teuthology.orchestra.run.smithi012.stderr: 3650K .......... .......... .......... .......... .......... 3% 4.39M 41s 2023-12-03T11:39:20.850 INFO:teuthology.orchestra.run.smithi012.stderr: 3700K .......... .......... .......... .......... .......... 4% 6.37M 41s 2023-12-03T11:39:20.860 INFO:teuthology.orchestra.run.smithi012.stderr: 3750K .......... .......... .......... .......... .......... 4% 4.71M 40s 2023-12-03T11:39:20.869 INFO:teuthology.orchestra.run.smithi012.stderr: 3800K .......... .......... .......... .......... .......... 4% 5.45M 40s 2023-12-03T11:39:20.881 INFO:teuthology.orchestra.run.smithi012.stderr: 3850K .......... .......... .......... .......... .......... 4% 5.18M 40s 2023-12-03T11:39:20.888 INFO:teuthology.orchestra.run.smithi012.stderr: 3900K .......... .......... .......... .......... .......... 4% 5.17M 39s 2023-12-03T11:39:20.899 INFO:teuthology.orchestra.run.smithi012.stderr: 3950K .......... .......... .......... .......... .......... 4% 5.83M 39s 2023-12-03T11:39:20.909 INFO:teuthology.orchestra.run.smithi012.stderr: 4000K .......... .......... .......... .......... .......... 4% 3.90M 39s 2023-12-03T11:39:20.919 INFO:teuthology.orchestra.run.smithi012.stderr: 4050K .......... .......... .......... .......... .......... 4% 5.05M 38s 2023-12-03T11:39:20.926 INFO:teuthology.orchestra.run.smithi012.stderr: 4100K .......... .......... .......... .......... .......... 4% 6.64M 38s 2023-12-03T11:39:20.936 INFO:teuthology.orchestra.run.smithi012.stderr: 4150K .......... .......... .......... .......... .......... 4% 5.06M 38s 2023-12-03T11:39:20.944 INFO:teuthology.orchestra.run.smithi012.stderr: 4200K .......... .......... .......... .......... .......... 4% 5.74M 38s 2023-12-03T11:39:20.954 INFO:teuthology.orchestra.run.smithi012.stderr: 4250K .......... .......... .......... .......... .......... 4% 5.04M 37s 2023-12-03T11:39:20.963 INFO:teuthology.orchestra.run.smithi012.stderr: 4300K .......... .......... .......... .......... .......... 4% 5.07M 37s 2023-12-03T11:39:20.974 INFO:teuthology.orchestra.run.smithi012.stderr: 4350K .......... .......... .......... .......... .......... 4% 6.27M 37s 2023-12-03T11:39:20.984 INFO:teuthology.orchestra.run.smithi012.stderr: 4400K .......... .......... .......... .......... .......... 4% 3.84M 37s 2023-12-03T11:39:20.993 INFO:teuthology.orchestra.run.smithi012.stderr: 4450K .......... .......... .......... .......... .......... 4% 5.32M 36s 2023-12-03T11:39:21.002 INFO:teuthology.orchestra.run.smithi012.stderr: 4500K .......... .......... .......... .......... .......... 4% 5.79M 36s 2023-12-03T11:39:21.010 INFO:teuthology.orchestra.run.smithi012.stderr: 4550K .......... .......... .......... .......... .......... 4% 5.83M 36s 2023-12-03T11:39:21.019 INFO:teuthology.orchestra.run.smithi012.stderr: 4600K .......... .......... .......... .......... .......... 5% 5.40M 36s 2023-12-03T11:39:21.030 INFO:teuthology.orchestra.run.smithi012.stderr: 4650K .......... .......... .......... .......... .......... 5% 5.75M 35s 2023-12-03T11:39:21.036 INFO:teuthology.orchestra.run.smithi012.stderr: 4700K .......... .......... .......... .......... .......... 5% 5.78M 35s 2023-12-03T11:39:21.047 INFO:teuthology.orchestra.run.smithi012.stderr: 4750K .......... .......... .......... .......... .......... 5% 6.28M 35s 2023-12-03T11:39:21.054 INFO:teuthology.orchestra.run.smithi012.stderr: 4800K .......... .......... .......... .......... .......... 5% 4.51M 35s 2023-12-03T11:39:21.064 INFO:teuthology.orchestra.run.smithi012.stderr: 4850K .......... .......... .......... .......... .......... 5% 4.84M 34s 2023-12-03T11:39:21.072 INFO:teuthology.orchestra.run.smithi012.stderr: 4900K .......... .......... .......... .......... .......... 5% 6.56M 34s 2023-12-03T11:39:21.081 INFO:teuthology.orchestra.run.smithi012.stderr: 4950K .......... .......... .......... .......... .......... 5% 5.19M 34s 2023-12-03T11:39:21.088 INFO:teuthology.orchestra.run.smithi012.stderr: 5000K .......... .......... .......... .......... .......... 5% 6.99M 34s 2023-12-03T11:39:21.097 INFO:teuthology.orchestra.run.smithi012.stderr: 5050K .......... .......... .......... .......... .......... 5% 5.52M 34s 2023-12-03T11:39:21.106 INFO:teuthology.orchestra.run.smithi012.stderr: 5100K .......... .......... .......... .......... .......... 5% 5.51M 33s 2023-12-03T11:39:21.117 INFO:teuthology.orchestra.run.smithi012.stderr: 5150K .......... .......... .......... .......... .......... 5% 6.12M 33s 2023-12-03T11:39:21.127 INFO:teuthology.orchestra.run.smithi012.stderr: 5200K .......... .......... .......... .......... .......... 5% 3.80M 33s 2023-12-03T11:39:21.133 INFO:teuthology.orchestra.run.smithi012.stderr: 5250K .......... .......... .......... .......... .......... 5% 7.35M 33s 2023-12-03T11:39:21.142 INFO:teuthology.orchestra.run.smithi012.stderr: 5300K .......... .......... .......... .......... .......... 5% 5.87M 33s 2023-12-03T11:39:21.152 INFO:teuthology.orchestra.run.smithi012.stderr: 5350K .......... .......... .......... .......... .......... 5% 4.63M 33s 2023-12-03T11:39:21.160 INFO:teuthology.orchestra.run.smithi012.stderr: 5400K .......... .......... .......... .......... .......... 5% 6.45M 32s 2023-12-03T11:39:21.170 INFO:teuthology.orchestra.run.smithi012.stderr: 5450K .......... .......... .......... .......... .......... 5% 4.66M 32s 2023-12-03T11:39:21.178 INFO:teuthology.orchestra.run.smithi012.stderr: 5500K .......... .......... .......... .......... .......... 5% 6.09M 32s 2023-12-03T11:39:21.191 INFO:teuthology.orchestra.run.smithi012.stderr: 5550K .......... .......... .......... .......... .......... 6% 5.25M 32s 2023-12-03T11:39:21.200 INFO:teuthology.orchestra.run.smithi012.stderr: 5600K .......... .......... .......... .......... .......... 6% 3.91M 32s 2023-12-03T11:39:21.209 INFO:teuthology.orchestra.run.smithi012.stderr: 5650K .......... .......... .......... .......... .......... 6% 5.34M 32s 2023-12-03T11:39:21.220 INFO:teuthology.orchestra.run.smithi012.stderr: 5700K .......... .......... .......... .......... .......... 6% 4.36M 31s 2023-12-03T11:39:21.230 INFO:teuthology.orchestra.run.smithi012.stderr: 5750K .......... .......... .......... .......... .......... 6% 5.26M 31s 2023-12-03T11:39:21.237 INFO:teuthology.orchestra.run.smithi012.stderr: 5800K .......... .......... .......... .......... .......... 6% 7.06M 31s 2023-12-03T11:39:21.245 INFO:teuthology.orchestra.run.smithi012.stderr: 5850K .......... .......... .......... .......... .......... 6% 6.03M 31s 2023-12-03T11:39:21.253 INFO:teuthology.orchestra.run.smithi012.stderr: 5900K .......... .......... .......... .......... .......... 6% 5.62M 31s 2023-12-03T11:39:21.265 INFO:teuthology.orchestra.run.smithi012.stderr: 5950K .......... .......... .......... .......... .......... 6% 6.43M 31s 2023-12-03T11:39:21.273 INFO:teuthology.orchestra.run.smithi012.stderr: 6000K .......... .......... .......... .......... .......... 6% 4.00M 31s 2023-12-03T11:39:21.281 INFO:teuthology.orchestra.run.smithi012.stderr: 6050K .......... .......... .......... .......... .......... 6% 5.98M 30s 2023-12-03T11:39:21.290 INFO:teuthology.orchestra.run.smithi012.stderr: 6100K .......... .......... .......... .......... .......... 6% 5.55M 30s 2023-12-03T11:39:21.296 INFO:teuthology.orchestra.run.smithi012.stderr: 6150K .......... .......... .......... .......... .......... 6% 7.80M 30s 2023-12-03T11:39:21.308 INFO:teuthology.orchestra.run.smithi012.stderr: 6200K .......... .......... .......... .......... .......... 6% 4.41M 30s 2023-12-03T11:39:21.314 INFO:teuthology.orchestra.run.smithi012.stderr: 6250K .......... .......... .......... .......... .......... 6% 7.42M 30s 2023-12-03T11:39:21.323 INFO:teuthology.orchestra.run.smithi012.stderr: 6300K .......... .......... .......... .......... .......... 6% 5.65M 30s 2023-12-03T11:39:21.332 INFO:teuthology.orchestra.run.smithi012.stderr: 6350K .......... .......... .......... .......... .......... 6% 7.65M 30s 2023-12-03T11:39:21.340 INFO:teuthology.orchestra.run.smithi012.stderr: 6400K .......... .......... .......... .......... .......... 6% 4.65M 29s 2023-12-03T11:39:21.346 INFO:teuthology.orchestra.run.smithi012.stderr: 6450K .......... .......... .......... .......... .......... 6% 7.52M 29s 2023-12-03T11:39:21.355 INFO:teuthology.orchestra.run.smithi012.stderr: 6500K .......... .......... .......... .......... .......... 7% 5.74M 29s 2023-12-03T11:39:21.361 INFO:teuthology.orchestra.run.smithi012.stderr: 6550K .......... .......... .......... .......... .......... 7% 7.64M 29s 2023-12-03T11:39:21.369 INFO:teuthology.orchestra.run.smithi012.stderr: 6600K .......... .......... .......... .......... .......... 7% 5.81M 29s 2023-12-03T11:39:21.376 INFO:teuthology.orchestra.run.smithi012.stderr: 6650K .......... .......... .......... .......... .......... 7% 7.29M 29s 2023-12-03T11:39:21.385 INFO:teuthology.orchestra.run.smithi012.stderr: 6700K .......... .......... .......... .......... .......... 7% 5.86M 29s 2023-12-03T11:39:21.392 INFO:teuthology.orchestra.run.smithi012.stderr: 6750K .......... .......... .......... .......... .......... 7% 8.86M 28s 2023-12-03T11:39:21.401 INFO:teuthology.orchestra.run.smithi012.stderr: 6800K .......... .......... .......... .......... .......... 7% 4.38M 28s 2023-12-03T11:39:21.407 INFO:teuthology.orchestra.run.smithi012.stderr: 6850K .......... .......... .......... .......... .......... 7% 8.49M 28s 2023-12-03T11:39:21.416 INFO:teuthology.orchestra.run.smithi012.stderr: 6900K .......... .......... .......... .......... .......... 7% 5.29M 28s 2023-12-03T11:39:21.421 INFO:teuthology.orchestra.run.smithi012.stderr: 6950K .......... .......... .......... .......... .......... 7% 9.85M 28s 2023-12-03T11:39:21.430 INFO:teuthology.orchestra.run.smithi012.stderr: 7000K .......... .......... .......... .......... .......... 7% 5.43M 28s 2023-12-03T11:39:21.436 INFO:teuthology.orchestra.run.smithi012.stderr: 7050K .......... .......... .......... .......... .......... 7% 8.19M 28s 2023-12-03T11:39:21.445 INFO:teuthology.orchestra.run.smithi012.stderr: 7100K .......... .......... .......... .......... .......... 7% 5.67M 28s 2023-12-03T11:39:21.452 INFO:teuthology.orchestra.run.smithi012.stderr: 7150K .......... .......... .......... .......... .......... 7% 8.71M 27s 2023-12-03T11:39:21.460 INFO:teuthology.orchestra.run.smithi012.stderr: 7200K .......... .......... .......... .......... .......... 7% 5.08M 27s 2023-12-03T11:39:21.466 INFO:teuthology.orchestra.run.smithi012.stderr: 7250K .......... .......... .......... .......... .......... 7% 8.03M 27s 2023-12-03T11:39:21.474 INFO:teuthology.orchestra.run.smithi012.stderr: 7300K .......... .......... .......... .......... .......... 7% 5.91M 27s 2023-12-03T11:39:21.480 INFO:teuthology.orchestra.run.smithi012.stderr: 7350K .......... .......... .......... .......... .......... 7% 8.56M 27s 2023-12-03T11:39:21.488 INFO:teuthology.orchestra.run.smithi012.stderr: 7400K .......... .......... .......... .......... .......... 8% 6.56M 27s 2023-12-03T11:39:21.494 INFO:teuthology.orchestra.run.smithi012.stderr: 7450K .......... .......... .......... .......... .......... 8% 7.41M 27s 2023-12-03T11:39:21.501 INFO:teuthology.orchestra.run.smithi012.stderr: 7500K .......... .......... .......... .......... .......... 8% 6.57M 27s 2023-12-03T11:39:21.510 INFO:teuthology.orchestra.run.smithi012.stderr: 7550K .......... .......... .......... .......... .......... 8% 7.12M 27s 2023-12-03T11:39:21.517 INFO:teuthology.orchestra.run.smithi012.stderr: 7600K .......... .......... .......... .......... .......... 8% 5.42M 26s 2023-12-03T11:39:21.524 INFO:teuthology.orchestra.run.smithi012.stderr: 7650K .......... .......... .......... .......... .......... 8% 7.44M 26s 2023-12-03T11:39:21.531 INFO:teuthology.orchestra.run.smithi012.stderr: 7700K .......... .......... .......... .......... .......... 8% 6.65M 26s 2023-12-03T11:39:21.538 INFO:teuthology.orchestra.run.smithi012.stderr: 7750K .......... .......... .......... .......... .......... 8% 7.45M 26s 2023-12-03T11:39:21.544 INFO:teuthology.orchestra.run.smithi012.stderr: 7800K .......... .......... .......... .......... .......... 8% 8.38M 26s 2023-12-03T11:39:21.552 INFO:teuthology.orchestra.run.smithi012.stderr: 7850K .......... .......... .......... .......... .......... 8% 6.11M 26s 2023-12-03T11:39:21.558 INFO:teuthology.orchestra.run.smithi012.stderr: 7900K .......... .......... .......... .......... .......... 8% 7.25M 26s 2023-12-03T11:39:21.568 INFO:teuthology.orchestra.run.smithi012.stderr: 7950K .......... .......... .......... .......... .......... 8% 6.87M 26s 2023-12-03T11:39:21.574 INFO:teuthology.orchestra.run.smithi012.stderr: 8000K .......... .......... .......... .......... .......... 8% 5.99M 26s 2023-12-03T11:39:21.582 INFO:teuthology.orchestra.run.smithi012.stderr: 8050K .......... .......... .......... .......... .......... 8% 6.09M 26s 2023-12-03T11:39:21.588 INFO:teuthology.orchestra.run.smithi012.stderr: 8100K .......... .......... .......... .......... .......... 8% 7.86M 25s 2023-12-03T11:39:21.595 INFO:teuthology.orchestra.run.smithi012.stderr: 8150K .......... .......... .......... .......... .......... 8% 6.61M 25s 2023-12-03T11:39:21.602 INFO:teuthology.orchestra.run.smithi012.stderr: 8200K .......... .......... .......... .......... .......... 8% 7.77M 25s 2023-12-03T11:39:21.608 INFO:teuthology.orchestra.run.smithi012.stderr: 8250K .......... .......... .......... .......... .......... 8% 7.08M 25s 2023-12-03T11:39:21.614 INFO:teuthology.orchestra.run.smithi012.stderr: 8300K .......... .......... .......... .......... .......... 8% 8.40M 25s 2023-12-03T11:39:21.623 INFO:teuthology.orchestra.run.smithi012.stderr: 8350K .......... .......... .......... .......... .......... 9% 8.06M 25s 2023-12-03T11:39:21.629 INFO:teuthology.orchestra.run.smithi012.stderr: 8400K .......... .......... .......... .......... .......... 9% 5.52M 25s 2023-12-03T11:39:21.636 INFO:teuthology.orchestra.run.smithi012.stderr: 8450K .......... .......... .......... .......... .......... 9% 7.02M 25s 2023-12-03T11:39:21.642 INFO:teuthology.orchestra.run.smithi012.stderr: 8500K .......... .......... .......... .......... .......... 9% 7.70M 25s 2023-12-03T11:39:21.651 INFO:teuthology.orchestra.run.smithi012.stderr: 8550K .......... .......... .......... .......... .......... 9% 5.77M 25s 2023-12-03T11:39:21.656 INFO:teuthology.orchestra.run.smithi012.stderr: 8600K .......... .......... .......... .......... .......... 9% 9.28M 24s 2023-12-03T11:39:21.662 INFO:teuthology.orchestra.run.smithi012.stderr: 8650K .......... .......... .......... .......... .......... 9% 8.43M 24s 2023-12-03T11:39:21.669 INFO:teuthology.orchestra.run.smithi012.stderr: 8700K .......... .......... .......... .......... .......... 9% 6.65M 24s 2023-12-03T11:39:21.677 INFO:teuthology.orchestra.run.smithi012.stderr: 8750K .......... .......... .......... .......... .......... 9% 8.31M 24s 2023-12-03T11:39:21.684 INFO:teuthology.orchestra.run.smithi012.stderr: 8800K .......... .......... .......... .......... .......... 9% 5.50M 24s 2023-12-03T11:39:21.690 INFO:teuthology.orchestra.run.smithi012.stderr: 8850K .......... .......... .......... .......... .......... 9% 7.89M 24s 2023-12-03T11:39:21.698 INFO:teuthology.orchestra.run.smithi012.stderr: 8900K .......... .......... .......... .......... .......... 9% 6.87M 24s 2023-12-03T11:39:21.704 INFO:teuthology.orchestra.run.smithi012.stderr: 8950K .......... .......... .......... .......... .......... 9% 7.81M 24s 2023-12-03T11:39:21.711 INFO:teuthology.orchestra.run.smithi012.stderr: 9000K .......... .......... .......... .......... .......... 9% 6.69M 24s 2023-12-03T11:39:21.717 INFO:teuthology.orchestra.run.smithi012.stderr: 9050K .......... .......... .......... .......... .......... 9% 8.06M 24s 2023-12-03T11:39:21.723 INFO:teuthology.orchestra.run.smithi012.stderr: 9100K .......... .......... .......... .......... .......... 9% 8.09M 24s 2023-12-03T11:39:21.731 INFO:teuthology.orchestra.run.smithi012.stderr: 9150K .......... .......... .......... .......... .......... 9% 8.17M 24s 2023-12-03T11:39:21.738 INFO:teuthology.orchestra.run.smithi012.stderr: 9200K .......... .......... .......... .......... .......... 9% 5.55M 23s 2023-12-03T11:39:21.743 INFO:teuthology.orchestra.run.smithi012.stderr: 9250K .......... .......... .......... .......... .......... 10% 8.36M 23s 2023-12-03T11:39:21.751 INFO:teuthology.orchestra.run.smithi012.stderr: 9300K .......... .......... .......... .......... .......... 10% 7.08M 23s 2023-12-03T11:39:21.758 INFO:teuthology.orchestra.run.smithi012.stderr: 9350K .......... .......... .......... .......... .......... 10% 7.00M 23s 2023-12-03T11:39:21.763 INFO:teuthology.orchestra.run.smithi012.stderr: 9400K .......... .......... .......... .......... .......... 10% 8.18M 23s 2023-12-03T11:39:21.770 INFO:teuthology.orchestra.run.smithi012.stderr: 9450K .......... .......... .......... .......... .......... 10% 6.84M 23s 2023-12-03T11:39:21.775 INFO:teuthology.orchestra.run.smithi012.stderr: 9500K .......... .......... .......... .......... .......... 10% 10.4M 23s 2023-12-03T11:39:21.783 INFO:teuthology.orchestra.run.smithi012.stderr: 9550K .......... .......... .......... .......... .......... 10% 6.97M 23s 2023-12-03T11:39:21.789 INFO:teuthology.orchestra.run.smithi012.stderr: 9600K .......... .......... .......... .......... .......... 10% 6.96M 23s 2023-12-03T11:39:21.796 INFO:teuthology.orchestra.run.smithi012.stderr: 9650K .......... .......... .......... .......... .......... 10% 7.51M 23s 2023-12-03T11:39:21.803 INFO:teuthology.orchestra.run.smithi012.stderr: 9700K .......... .......... .......... .......... .......... 10% 6.81M 23s 2023-12-03T11:39:21.808 INFO:teuthology.orchestra.run.smithi012.stderr: 9750K .......... .......... .......... .......... .......... 10% 9.59M 23s 2023-12-03T11:39:21.814 INFO:teuthology.orchestra.run.smithi012.stderr: 9800K .......... .......... .......... .......... .......... 10% 8.59M 22s 2023-12-03T11:39:21.820 INFO:teuthology.orchestra.run.smithi012.stderr: 9850K .......... .......... .......... .......... .......... 10% 7.33M 22s 2023-12-03T11:39:21.825 INFO:teuthology.orchestra.run.smithi012.stderr: 9900K .......... .......... .......... .......... .......... 10% 9.54M 22s 2023-12-03T11:39:21.835 INFO:teuthology.orchestra.run.smithi012.stderr: 9950K .......... .......... .......... .......... .......... 10% 6.64M 22s 2023-12-03T11:39:21.840 INFO:teuthology.orchestra.run.smithi012.stderr: 10000K .......... .......... .......... .......... .......... 10% 6.69M 22s 2023-12-03T11:39:21.848 INFO:teuthology.orchestra.run.smithi012.stderr: 10050K .......... .......... .......... .......... .......... 10% 6.64M 22s 2023-12-03T11:39:21.853 INFO:teuthology.orchestra.run.smithi012.stderr: 10100K .......... .......... .......... .......... .......... 10% 9.61M 22s 2023-12-03T11:39:21.860 INFO:teuthology.orchestra.run.smithi012.stderr: 10150K .......... .......... .......... .......... .......... 10% 6.15M 22s 2023-12-03T11:39:21.865 INFO:teuthology.orchestra.run.smithi012.stderr: 10200K .......... .......... .......... .......... .......... 11% 10.3M 22s 2023-12-03T11:39:21.873 INFO:teuthology.orchestra.run.smithi012.stderr: 10250K .......... .......... .......... .......... .......... 11% 6.51M 22s 2023-12-03T11:39:21.878 INFO:teuthology.orchestra.run.smithi012.stderr: 10300K .......... .......... .......... .......... .......... 11% 9.84M 22s 2023-12-03T11:39:21.887 INFO:teuthology.orchestra.run.smithi012.stderr: 10350K .......... .......... .......... .......... .......... 11% 10.5M 22s 2023-12-03T11:39:21.892 INFO:teuthology.orchestra.run.smithi012.stderr: 10400K .......... .......... .......... .......... .......... 11% 5.15M 22s 2023-12-03T11:39:21.897 INFO:teuthology.orchestra.run.smithi012.stderr: 10450K .......... .......... .......... .......... .......... 11% 9.36M 22s 2023-12-03T11:39:21.904 INFO:teuthology.orchestra.run.smithi012.stderr: 10500K .......... .......... .......... .......... .......... 11% 6.64M 22s 2023-12-03T11:39:21.910 INFO:teuthology.orchestra.run.smithi012.stderr: 10550K .......... .......... .......... .......... .......... 11% 8.90M 21s 2023-12-03T11:39:21.918 INFO:teuthology.orchestra.run.smithi012.stderr: 10600K .......... .......... .......... .......... .......... 11% 6.22M 21s 2023-12-03T11:39:21.922 INFO:teuthology.orchestra.run.smithi012.stderr: 10650K .......... .......... .......... .......... .......... 11% 11.2M 21s 2023-12-03T11:39:21.926 INFO:teuthology.orchestra.run.smithi012.stderr: 10700K .......... .......... .......... .......... .......... 11% 11.4M 21s 2023-12-03T11:39:21.935 INFO:teuthology.orchestra.run.smithi012.stderr: 10750K .......... .......... .......... .......... .......... 11% 6.00M 21s 2023-12-03T11:39:21.941 INFO:teuthology.orchestra.run.smithi012.stderr: 10800K .......... .......... .......... .......... .......... 11% 7.77M 21s 2023-12-03T11:39:21.948 INFO:teuthology.orchestra.run.smithi012.stderr: 10850K .......... .......... .......... .......... .......... 11% 6.59M 21s 2023-12-03T11:39:21.953 INFO:teuthology.orchestra.run.smithi012.stderr: 10900K .......... .......... .......... .......... .......... 11% 11.0M 21s 2023-12-03T11:39:21.957 INFO:teuthology.orchestra.run.smithi012.stderr: 10950K .......... .......... .......... .......... .......... 11% 11.1M 21s 2023-12-03T11:39:21.966 INFO:teuthology.orchestra.run.smithi012.stderr: 11000K .......... .......... .......... .......... .......... 11% 5.86M 21s 2023-12-03T11:39:21.970 INFO:teuthology.orchestra.run.smithi012.stderr: 11050K .......... .......... .......... .......... .......... 11% 11.4M 21s 2023-12-03T11:39:21.977 INFO:teuthology.orchestra.run.smithi012.stderr: 11100K .......... .......... .......... .......... .......... 12% 7.09M 21s 2023-12-03T11:39:21.983 INFO:teuthology.orchestra.run.smithi012.stderr: 11150K .......... .......... .......... .......... .......... 12% 9.42M 21s 2023-12-03T11:39:21.988 INFO:teuthology.orchestra.run.smithi012.stderr: 11200K .......... .......... .......... .......... .......... 12% 8.01M 21s 2023-12-03T11:39:21.995 INFO:teuthology.orchestra.run.smithi012.stderr: 11250K .......... .......... .......... .......... .......... 12% 6.64M 21s 2023-12-03T11:39:21.999 INFO:teuthology.orchestra.run.smithi012.stderr: 11300K .......... .......... .......... .......... .......... 12% 11.5M 21s 2023-12-03T11:39:22.006 INFO:teuthology.orchestra.run.smithi012.stderr: 11350K .......... .......... .......... .......... .......... 12% 7.13M 20s 2023-12-03T11:39:22.012 INFO:teuthology.orchestra.run.smithi012.stderr: 11400K .......... .......... .......... .......... .......... 12% 9.49M 20s 2023-12-03T11:39:22.016 INFO:teuthology.orchestra.run.smithi012.stderr: 11450K .......... .......... .......... .......... .......... 12% 11.3M 20s 2023-12-03T11:39:22.023 INFO:teuthology.orchestra.run.smithi012.stderr: 11500K .......... .......... .......... .......... .......... 12% 6.66M 20s 2023-12-03T11:39:22.029 INFO:teuthology.orchestra.run.smithi012.stderr: 11550K .......... .......... .......... .......... .......... 12% 12.2M 20s 2023-12-03T11:39:22.037 INFO:teuthology.orchestra.run.smithi012.stderr: 11600K .......... .......... .......... .......... .......... 12% 5.13M 20s 2023-12-03T11:39:22.041 INFO:teuthology.orchestra.run.smithi012.stderr: 11650K .......... .......... .......... .......... .......... 12% 11.0M 20s 2023-12-03T11:39:22.046 INFO:teuthology.orchestra.run.smithi012.stderr: 11700K .......... .......... .......... .......... .......... 12% 10.6M 20s 2023-12-03T11:39:22.053 INFO:teuthology.orchestra.run.smithi012.stderr: 11750K .......... .......... .......... .......... .......... 12% 6.71M 20s 2023-12-03T11:39:22.058 INFO:teuthology.orchestra.run.smithi012.stderr: 11800K .......... .......... .......... .......... .......... 12% 9.23M 20s 2023-12-03T11:39:22.063 INFO:teuthology.orchestra.run.smithi012.stderr: 11850K .......... .......... .......... .......... .......... 12% 11.1M 20s 2023-12-03T11:39:22.070 INFO:teuthology.orchestra.run.smithi012.stderr: 11900K .......... .......... .......... .......... .......... 12% 6.89M 20s 2023-12-03T11:39:22.076 INFO:teuthology.orchestra.run.smithi012.stderr: 11950K .......... .......... .......... .......... .......... 12% 12.7M 20s 2023-12-03T11:39:22.082 INFO:teuthology.orchestra.run.smithi012.stderr: 12000K .......... .......... .......... .......... .......... 12% 5.48M 20s 2023-12-03T11:39:22.088 INFO:teuthology.orchestra.run.smithi012.stderr: 12050K .......... .......... .......... .......... .......... 13% 9.10M 20s 2023-12-03T11:39:22.093 INFO:teuthology.orchestra.run.smithi012.stderr: 12100K .......... .......... .......... .......... .......... 13% 10.6M 20s 2023-12-03T11:39:22.100 INFO:teuthology.orchestra.run.smithi012.stderr: 12150K .......... .......... .......... .......... .......... 13% 6.34M 20s 2023-12-03T11:39:22.105 INFO:teuthology.orchestra.run.smithi012.stderr: 12200K .......... .......... .......... .......... .......... 13% 11.0M 19s 2023-12-03T11:39:22.111 INFO:teuthology.orchestra.run.smithi012.stderr: 12250K .......... .......... .......... .......... .......... 13% 7.42M 19s 2023-12-03T11:39:22.116 INFO:teuthology.orchestra.run.smithi012.stderr: 12300K .......... .......... .......... .......... .......... 13% 10.4M 19s 2023-12-03T11:39:22.122 INFO:teuthology.orchestra.run.smithi012.stderr: 12350K .......... .......... .......... .......... .......... 13% 9.65M 19s 2023-12-03T11:39:22.130 INFO:teuthology.orchestra.run.smithi012.stderr: 12400K .......... .......... .......... .......... .......... 13% 5.45M 19s 2023-12-03T11:39:22.134 INFO:teuthology.orchestra.run.smithi012.stderr: 12450K .......... .......... .......... .......... .......... 13% 11.3M 19s 2023-12-03T11:39:22.140 INFO:teuthology.orchestra.run.smithi012.stderr: 12500K .......... .......... .......... .......... .......... 13% 8.05M 19s 2023-12-03T11:39:22.146 INFO:teuthology.orchestra.run.smithi012.stderr: 12550K .......... .......... .......... .......... .......... 13% 8.21M 19s 2023-12-03T11:39:22.151 INFO:teuthology.orchestra.run.smithi012.stderr: 12600K .......... .......... .......... .......... .......... 13% 10.6M 19s 2023-12-03T11:39:22.156 INFO:teuthology.orchestra.run.smithi012.stderr: 12650K .......... .......... .......... .......... .......... 13% 10.0M 19s 2023-12-03T11:39:22.161 INFO:teuthology.orchestra.run.smithi012.stderr: 12700K .......... .......... .......... .......... .......... 13% 8.46M 19s 2023-12-03T11:39:22.167 INFO:teuthology.orchestra.run.smithi012.stderr: 12750K .......... .......... .......... .......... .......... 13% 9.52M 19s 2023-12-03T11:39:22.175 INFO:teuthology.orchestra.run.smithi012.stderr: 12800K .......... .......... .......... .......... .......... 13% 6.16M 19s 2023-12-03T11:39:22.179 INFO:teuthology.orchestra.run.smithi012.stderr: 12850K .......... .......... .......... .......... .......... 13% 10.5M 19s 2023-12-03T11:39:22.183 INFO:teuthology.orchestra.run.smithi012.stderr: 12900K .......... .......... .......... .......... .......... 13% 11.9M 19s 2023-12-03T11:39:22.191 INFO:teuthology.orchestra.run.smithi012.stderr: 12950K .......... .......... .......... .......... .......... 13% 6.45M 19s 2023-12-03T11:39:22.197 INFO:teuthology.orchestra.run.smithi012.stderr: 13000K .......... .......... .......... .......... .......... 14% 8.29M 19s 2023-12-03T11:39:22.201 INFO:teuthology.orchestra.run.smithi012.stderr: 13050K .......... .......... .......... .......... .......... 14% 11.9M 19s 2023-12-03T11:39:22.206 INFO:teuthology.orchestra.run.smithi012.stderr: 13100K .......... .......... .......... .......... .......... 14% 8.87M 19s 2023-12-03T11:39:22.214 INFO:teuthology.orchestra.run.smithi012.stderr: 13150K .......... .......... .......... .......... .......... 14% 8.89M 19s 2023-12-03T11:39:22.219 INFO:teuthology.orchestra.run.smithi012.stderr: 13200K .......... .......... .......... .......... .......... 14% 7.12M 18s 2023-12-03T11:39:22.225 INFO:teuthology.orchestra.run.smithi012.stderr: 13250K .......... .......... .......... .......... .......... 14% 7.55M 18s 2023-12-03T11:39:22.229 INFO:teuthology.orchestra.run.smithi012.stderr: 13300K .......... .......... .......... .......... .......... 14% 12.5M 18s 2023-12-03T11:39:22.234 INFO:teuthology.orchestra.run.smithi012.stderr: 13350K .......... .......... .......... .......... .......... 14% 10.0M 18s 2023-12-03T11:39:22.240 INFO:teuthology.orchestra.run.smithi012.stderr: 13400K .......... .......... .......... .......... .......... 14% 8.16M 18s 2023-12-03T11:39:22.245 INFO:teuthology.orchestra.run.smithi012.stderr: 13450K .......... .......... .......... .......... .......... 14% 10.1M 18s 2023-12-03T11:39:22.249 INFO:teuthology.orchestra.run.smithi012.stderr: 13500K .......... .......... .......... .......... .......... 14% 11.2M 18s 2023-12-03T11:39:22.257 INFO:teuthology.orchestra.run.smithi012.stderr: 13550K .......... .......... .......... .......... .......... 14% 7.23M 18s 2023-12-03T11:39:22.262 INFO:teuthology.orchestra.run.smithi012.stderr: 13600K .......... .......... .......... .......... .......... 14% 8.34M 18s 2023-12-03T11:39:22.268 INFO:teuthology.orchestra.run.smithi012.stderr: 13650K .......... .......... .......... .......... .......... 14% 7.51M 18s 2023-12-03T11:39:22.273 INFO:teuthology.orchestra.run.smithi012.stderr: 13700K .......... .......... .......... .......... .......... 14% 10.0M 18s 2023-12-03T11:39:22.276 INFO:teuthology.orchestra.run.smithi012.stderr: 13750K .......... .......... .......... .......... .......... 14% 14.4M 18s 2023-12-03T11:39:22.284 INFO:teuthology.orchestra.run.smithi012.stderr: 13800K .......... .......... .......... .......... .......... 14% 6.69M 18s 2023-12-03T11:39:22.288 INFO:teuthology.orchestra.run.smithi012.stderr: 13850K .......... .......... .......... .......... .......... 14% 11.8M 18s 2023-12-03T11:39:22.292 INFO:teuthology.orchestra.run.smithi012.stderr: 13900K .......... .......... .......... .......... .......... 15% 10.6M 18s 2023-12-03T11:39:22.301 INFO:teuthology.orchestra.run.smithi012.stderr: 13950K .......... .......... .......... .......... .......... 15% 8.12M 18s 2023-12-03T11:39:22.305 INFO:teuthology.orchestra.run.smithi012.stderr: 14000K .......... .......... .......... .......... .......... 15% 8.40M 18s 2023-12-03T11:39:22.311 INFO:teuthology.orchestra.run.smithi012.stderr: 14050K .......... .......... .......... .......... .......... 15% 7.39M 18s 2023-12-03T11:39:22.315 INFO:teuthology.orchestra.run.smithi012.stderr: 14100K .......... .......... .......... .......... .......... 15% 11.1M 18s 2023-12-03T11:39:22.321 INFO:teuthology.orchestra.run.smithi012.stderr: 14150K .......... .......... .......... .......... .......... 15% 9.28M 18s 2023-12-03T11:39:22.329 INFO:teuthology.orchestra.run.smithi012.stderr: 14200K .......... .......... .......... .......... .......... 15% 6.31M 18s 2023-12-03T11:39:22.331 INFO:teuthology.orchestra.run.smithi012.stderr: 14250K .......... .......... .......... .......... .......... 15% 16.4M 18s 2023-12-03T11:39:22.336 INFO:teuthology.orchestra.run.smithi012.stderr: 14300K .......... .......... .......... .......... .......... 15% 11.0M 17s 2023-12-03T11:39:22.345 INFO:teuthology.orchestra.run.smithi012.stderr: 14350K .......... .......... .......... .......... .......... 15% 6.59M 17s 2023-12-03T11:39:22.348 INFO:teuthology.orchestra.run.smithi012.stderr: 14400K .......... .......... .......... .......... .......... 15% 9.92M 17s 2023-12-03T11:39:22.353 INFO:teuthology.orchestra.run.smithi012.stderr: 14450K .......... .......... .......... .......... .......... 15% 10.4M 17s 2023-12-03T11:39:22.360 INFO:teuthology.orchestra.run.smithi012.stderr: 14500K .......... .......... .......... .......... .......... 15% 7.38M 17s 2023-12-03T11:39:22.364 INFO:teuthology.orchestra.run.smithi012.stderr: 14550K .......... .......... .......... .......... .......... 15% 10.6M 17s 2023-12-03T11:39:22.367 INFO:teuthology.orchestra.run.smithi012.stderr: 14600K .......... .......... .......... .......... .......... 15% 13.9M 17s 2023-12-03T11:39:22.375 INFO:teuthology.orchestra.run.smithi012.stderr: 14650K .......... .......... .......... .......... .......... 15% 6.41M 17s 2023-12-03T11:39:22.378 INFO:teuthology.orchestra.run.smithi012.stderr: 14700K .......... .......... .......... .......... .......... 15% 15.2M 17s 2023-12-03T11:39:22.386 INFO:teuthology.orchestra.run.smithi012.stderr: 14750K .......... .......... .......... .......... .......... 15% 6.93M 17s 2023-12-03T11:39:22.392 INFO:teuthology.orchestra.run.smithi012.stderr: 14800K .......... .......... .......... .......... .......... 15% 7.92M 17s 2023-12-03T11:39:22.395 INFO:teuthology.orchestra.run.smithi012.stderr: 14850K .......... .......... .......... .......... .......... 16% 13.1M 17s 2023-12-03T11:39:22.401 INFO:teuthology.orchestra.run.smithi012.stderr: 14900K .......... .......... .......... .......... .......... 16% 8.58M 17s 2023-12-03T11:39:22.406 INFO:teuthology.orchestra.run.smithi012.stderr: 14950K .......... .......... .......... .......... .......... 16% 10.4M 17s 2023-12-03T11:39:22.410 INFO:teuthology.orchestra.run.smithi012.stderr: 15000K .......... .......... .......... .......... .......... 16% 10.3M 17s 2023-12-03T11:39:22.415 INFO:teuthology.orchestra.run.smithi012.stderr: 15050K .......... .......... .......... .......... .......... 16% 9.84M 17s 2023-12-03T11:39:22.421 INFO:teuthology.orchestra.run.smithi012.stderr: 15100K .......... .......... .......... .......... .......... 16% 8.30M 17s 2023-12-03T11:39:22.430 INFO:teuthology.orchestra.run.smithi012.stderr: 15150K .......... .......... .......... .......... .......... 16% 6.20M 17s 2023-12-03T11:39:22.435 INFO:teuthology.orchestra.run.smithi012.stderr: 15200K .......... .......... .......... .......... .......... 16% 8.41M 17s 2023-12-03T11:39:22.438 INFO:teuthology.orchestra.run.smithi012.stderr: 15250K .......... .......... .......... .......... .......... 16% 15.6M 17s 2023-12-03T11:39:22.445 INFO:teuthology.orchestra.run.smithi012.stderr: 15300K .......... .......... .......... .......... .......... 16% 7.52M 17s 2023-12-03T11:39:22.449 INFO:teuthology.orchestra.run.smithi012.stderr: 15350K .......... .......... .......... .......... .......... 16% 12.5M 17s 2023-12-03T11:39:22.453 INFO:teuthology.orchestra.run.smithi012.stderr: 15400K .......... .......... .......... .......... .......... 16% 10.9M 17s 2023-12-03T11:39:22.459 INFO:teuthology.orchestra.run.smithi012.stderr: 15450K .......... .......... .......... .......... .......... 16% 8.13M 17s 2023-12-03T11:39:22.463 INFO:teuthology.orchestra.run.smithi012.stderr: 15500K .......... .......... .......... .......... .......... 16% 11.3M 17s 2023-12-03T11:39:22.469 INFO:teuthology.orchestra.run.smithi012.stderr: 15550K .......... .......... .......... .......... .......... 16% 12.4M 16s 2023-12-03T11:39:22.474 INFO:teuthology.orchestra.run.smithi012.stderr: 15600K .......... .......... .......... .......... .......... 16% 6.95M 16s 2023-12-03T11:39:22.483 INFO:teuthology.orchestra.run.smithi012.stderr: 15650K .......... .......... .......... .......... .......... 16% 6.30M 16s 2023-12-03T11:39:22.483 INFO:teuthology.orchestra.run.smithi012.stderr: 15700K .......... .......... .......... .......... .......... 16% 74.9M 16s 2023-12-03T11:39:22.489 INFO:teuthology.orchestra.run.smithi012.stderr: 15750K .......... .......... .......... .......... .......... 17% 7.22M 16s 2023-12-03T11:39:22.493 INFO:teuthology.orchestra.run.smithi012.stderr: 15800K .......... .......... .......... .......... .......... 17% 12.7M 16s 2023-12-03T11:39:22.500 INFO:teuthology.orchestra.run.smithi012.stderr: 15850K .......... .......... .......... .......... .......... 17% 7.10M 16s 2023-12-03T11:39:22.504 INFO:teuthology.orchestra.run.smithi012.stderr: 15900K .......... .......... .......... .......... .......... 17% 13.5M 16s 2023-12-03T11:39:22.513 INFO:teuthology.orchestra.run.smithi012.stderr: 15950K .......... .......... .......... .......... .......... 17% 11.0M 16s 2023-12-03T11:39:22.516 INFO:teuthology.orchestra.run.smithi012.stderr: 16000K .......... .......... .......... .......... .......... 17% 6.34M 16s 2023-12-03T11:39:22.520 INFO:teuthology.orchestra.run.smithi012.stderr: 16050K .......... .......... .......... .......... .......... 17% 11.0M 16s 2023-12-03T11:39:22.527 INFO:teuthology.orchestra.run.smithi012.stderr: 16100K .......... .......... .......... .......... .......... 17% 7.74M 16s 2023-12-03T11:39:22.531 INFO:teuthology.orchestra.run.smithi012.stderr: 16150K .......... .......... .......... .......... .......... 17% 11.2M 16s 2023-12-03T11:39:22.534 INFO:teuthology.orchestra.run.smithi012.stderr: 16200K .......... .......... .......... .......... .......... 17% 15.0M 16s 2023-12-03T11:39:22.543 INFO:teuthology.orchestra.run.smithi012.stderr: 16250K .......... .......... .......... .......... .......... 17% 6.01M 16s 2023-12-03T11:39:22.546 INFO:teuthology.orchestra.run.smithi012.stderr: 16300K .......... .......... .......... .......... .......... 17% 13.5M 16s 2023-12-03T11:39:22.551 INFO:teuthology.orchestra.run.smithi012.stderr: 16350K .......... .......... .......... .......... .......... 17% 10.9M 16s 2023-12-03T11:39:22.558 INFO:teuthology.orchestra.run.smithi012.stderr: 16400K .......... .......... .......... .......... .......... 17% 6.86M 16s 2023-12-03T11:39:22.562 INFO:teuthology.orchestra.run.smithi012.stderr: 16450K .......... .......... .......... .......... .......... 17% 11.8M 16s 2023-12-03T11:39:22.566 INFO:teuthology.orchestra.run.smithi012.stderr: 16500K .......... .......... .......... .......... .......... 17% 10.7M 16s 2023-12-03T11:39:22.572 INFO:teuthology.orchestra.run.smithi012.stderr: 16550K .......... .......... .......... .......... .......... 17% 8.26M 16s 2023-12-03T11:39:22.576 INFO:teuthology.orchestra.run.smithi012.stderr: 16600K .......... .......... .......... .......... .......... 17% 11.5M 16s 2023-12-03T11:39:22.580 INFO:teuthology.orchestra.run.smithi012.stderr: 16650K .......... .......... .......... .......... .......... 17% 12.0M 16s 2023-12-03T11:39:22.588 INFO:teuthology.orchestra.run.smithi012.stderr: 16700K .......... .......... .......... .......... .......... 18% 6.79M 16s 2023-12-03T11:39:22.593 INFO:teuthology.orchestra.run.smithi012.stderr: 16750K .......... .......... .......... .......... .......... 18% 17.2M 16s 2023-12-03T11:39:22.595 INFO:teuthology.orchestra.run.smithi012.stderr: 16800K .......... .......... .......... .......... .......... 18% 10.0M 16s 2023-12-03T11:39:22.602 INFO:teuthology.orchestra.run.smithi012.stderr: 16850K .......... .......... .......... .......... .......... 18% 6.98M 16s 2023-12-03T11:39:22.606 INFO:teuthology.orchestra.run.smithi012.stderr: 16900K .......... .......... .......... .......... .......... 18% 13.6M 16s 2023-12-03T11:39:22.610 INFO:teuthology.orchestra.run.smithi012.stderr: 16950K .......... .......... .......... .......... .......... 18% 11.4M 15s 2023-12-03T11:39:22.614 INFO:teuthology.orchestra.run.smithi012.stderr: 17000K .......... .......... .......... .......... .......... 18% 12.8M 15s 2023-12-03T11:39:22.620 INFO:teuthology.orchestra.run.smithi012.stderr: 17050K .......... .......... .......... .......... .......... 18% 8.08M 15s 2023-12-03T11:39:22.625 INFO:teuthology.orchestra.run.smithi012.stderr: 17100K .......... .......... .......... .......... .......... 18% 11.1M 15s 2023-12-03T11:39:22.635 INFO:teuthology.orchestra.run.smithi012.stderr: 17150K .......... .......... .......... .......... .......... 18% 10.6M 15s 2023-12-03T11:39:22.637 INFO:teuthology.orchestra.run.smithi012.stderr: 17200K .......... .......... .......... .......... .......... 18% 6.41M 15s 2023-12-03T11:39:22.640 INFO:teuthology.orchestra.run.smithi012.stderr: 17250K .......... .......... .......... .......... .......... 18% 16.4M 15s 2023-12-03T11:39:22.644 INFO:teuthology.orchestra.run.smithi012.stderr: 17300K .......... .......... .......... .......... .......... 18% 11.4M 15s 2023-12-03T11:39:22.653 INFO:teuthology.orchestra.run.smithi012.stderr: 17350K .......... .......... .......... .......... .......... 18% 5.64M 15s 2023-12-03T11:39:22.656 INFO:teuthology.orchestra.run.smithi012.stderr: 17400K .......... .......... .......... .......... .......... 18% 13.8M 15s 2023-12-03T11:39:22.660 INFO:teuthology.orchestra.run.smithi012.stderr: 17450K .......... .......... .......... .......... .......... 18% 12.0M 15s 2023-12-03T11:39:22.666 INFO:teuthology.orchestra.run.smithi012.stderr: 17500K .......... .......... .......... .......... .......... 18% 8.52M 15s 2023-12-03T11:39:22.671 INFO:teuthology.orchestra.run.smithi012.stderr: 17550K .......... .......... .......... .......... .......... 18% 11.4M 15s 2023-12-03T11:39:22.675 INFO:teuthology.orchestra.run.smithi012.stderr: 17600K .......... .......... .......... .......... .......... 18% 10.5M 15s 2023-12-03T11:39:22.682 INFO:teuthology.orchestra.run.smithi012.stderr: 17650K .......... .......... .......... .......... .......... 19% 6.95M 15s 2023-12-03T11:39:22.686 INFO:teuthology.orchestra.run.smithi012.stderr: 17700K .......... .......... .......... .......... .......... 19% 12.4M 15s 2023-12-03T11:39:22.689 INFO:teuthology.orchestra.run.smithi012.stderr: 17750K .......... .......... .......... .......... .......... 19% 13.9M 15s 2023-12-03T11:39:22.696 INFO:teuthology.orchestra.run.smithi012.stderr: 17800K .......... .......... .......... .......... .......... 19% 7.51M 15s 2023-12-03T11:39:22.700 INFO:teuthology.orchestra.run.smithi012.stderr: 17850K .......... .......... .......... .......... .......... 19% 12.1M 15s 2023-12-03T11:39:22.704 INFO:teuthology.orchestra.run.smithi012.stderr: 17900K .......... .......... .......... .......... .......... 19% 11.0M 15s 2023-12-03T11:39:22.712 INFO:teuthology.orchestra.run.smithi012.stderr: 17950K .......... .......... .......... .......... .......... 19% 8.58M 15s 2023-12-03T11:39:22.716 INFO:teuthology.orchestra.run.smithi012.stderr: 18000K .......... .......... .......... .......... .......... 19% 8.63M 15s 2023-12-03T11:39:22.719 INFO:teuthology.orchestra.run.smithi012.stderr: 18050K .......... .......... .......... .......... .......... 19% 16.9M 15s 2023-12-03T11:39:22.726 INFO:teuthology.orchestra.run.smithi012.stderr: 18100K .......... .......... .......... .......... .......... 19% 6.58M 15s 2023-12-03T11:39:22.730 INFO:teuthology.orchestra.run.smithi012.stderr: 18150K .......... .......... .......... .......... .......... 19% 12.7M 15s 2023-12-03T11:39:22.734 INFO:teuthology.orchestra.run.smithi012.stderr: 18200K .......... .......... .......... .......... .......... 19% 13.3M 15s 2023-12-03T11:39:22.740 INFO:teuthology.orchestra.run.smithi012.stderr: 18250K .......... .......... .......... .......... .......... 19% 7.92M 15s 2023-12-03T11:39:22.744 INFO:teuthology.orchestra.run.smithi012.stderr: 18300K .......... .......... .......... .......... .......... 19% 13.1M 15s 2023-12-03T11:39:22.748 INFO:teuthology.orchestra.run.smithi012.stderr: 18350K .......... .......... .......... .......... .......... 19% 12.2M 15s 2023-12-03T11:39:22.754 INFO:teuthology.orchestra.run.smithi012.stderr: 18400K .......... .......... .......... .......... .......... 19% 6.98M 15s 2023-12-03T11:39:22.759 INFO:teuthology.orchestra.run.smithi012.stderr: 18450K .......... .......... .......... .......... .......... 19% 10.9M 15s 2023-12-03T11:39:22.762 INFO:teuthology.orchestra.run.smithi012.stderr: 18500K .......... .......... .......... .......... .......... 19% 14.2M 14s 2023-12-03T11:39:22.765 INFO:teuthology.orchestra.run.smithi012.stderr: 18550K .......... .......... .......... .......... .......... 20% 16.7M 14s 2023-12-03T11:39:22.772 INFO:teuthology.orchestra.run.smithi012.stderr: 18600K .......... .......... .......... .......... .......... 20% 7.01M 14s 2023-12-03T11:39:22.776 INFO:teuthology.orchestra.run.smithi012.stderr: 18650K .......... .......... .......... .......... .......... 20% 12.3M 14s 2023-12-03T11:39:22.780 INFO:teuthology.orchestra.run.smithi012.stderr: 18700K .......... .......... .......... .......... .......... 20% 13.1M 14s 2023-12-03T11:39:22.788 INFO:teuthology.orchestra.run.smithi012.stderr: 18750K .......... .......... .......... .......... .......... 20% 8.41M 14s 2023-12-03T11:39:22.791 INFO:teuthology.orchestra.run.smithi012.stderr: 18800K .......... .......... .......... .......... .......... 20% 8.86M 14s 2023-12-03T11:39:22.795 INFO:teuthology.orchestra.run.smithi012.stderr: 18850K .......... .......... .......... .......... .......... 20% 14.9M 14s 2023-12-03T11:39:22.801 INFO:teuthology.orchestra.run.smithi012.stderr: 18900K .......... .......... .......... .......... .......... 20% 7.99M 14s 2023-12-03T11:39:22.804 INFO:teuthology.orchestra.run.smithi012.stderr: 18950K .......... .......... .......... .......... .......... 20% 13.6M 14s 2023-12-03T11:39:22.808 INFO:teuthology.orchestra.run.smithi012.stderr: 19000K .......... .......... .......... .......... .......... 20% 12.9M 14s 2023-12-03T11:39:22.813 INFO:teuthology.orchestra.run.smithi012.stderr: 19050K .......... .......... .......... .......... .......... 20% 9.38M 14s 2023-12-03T11:39:22.818 INFO:teuthology.orchestra.run.smithi012.stderr: 19100K .......... .......... .......... .......... .......... 20% 11.1M 14s 2023-12-03T11:39:22.822 INFO:teuthology.orchestra.run.smithi012.stderr: 19150K .......... .......... .......... .......... .......... 20% 14.2M 14s 2023-12-03T11:39:22.828 INFO:teuthology.orchestra.run.smithi012.stderr: 19200K .......... .......... .......... .......... .......... 20% 7.16M 14s 2023-12-03T11:39:22.832 INFO:teuthology.orchestra.run.smithi012.stderr: 19250K .......... .......... .......... .......... .......... 20% 11.0M 14s 2023-12-03T11:39:22.835 INFO:teuthology.orchestra.run.smithi012.stderr: 19300K .......... .......... .......... .......... .......... 20% 18.7M 14s 2023-12-03T11:39:22.839 INFO:teuthology.orchestra.run.smithi012.stderr: 19350K .......... .......... .......... .......... .......... 20% 11.8M 14s 2023-12-03T11:39:22.846 INFO:teuthology.orchestra.run.smithi012.stderr: 19400K .......... .......... .......... .......... .......... 20% 7.43M 14s 2023-12-03T11:39:22.850 INFO:teuthology.orchestra.run.smithi012.stderr: 19450K .......... .......... .......... .......... .......... 20% 13.3M 14s 2023-12-03T11:39:22.853 INFO:teuthology.orchestra.run.smithi012.stderr: 19500K .......... .......... .......... .......... .......... 21% 13.3M 14s 2023-12-03T11:39:22.861 INFO:teuthology.orchestra.run.smithi012.stderr: 19550K .......... .......... .......... .......... .......... 21% 8.43M 14s 2023-12-03T11:39:22.864 INFO:teuthology.orchestra.run.smithi012.stderr: 19600K .......... .......... .......... .......... .......... 21% 10.2M 14s 2023-12-03T11:39:22.868 INFO:teuthology.orchestra.run.smithi012.stderr: 19650K .......... .......... .......... .......... .......... 21% 13.2M 14s 2023-12-03T11:39:22.872 INFO:teuthology.orchestra.run.smithi012.stderr: 19700K .......... .......... .......... .......... .......... 21% 9.83M 14s 2023-12-03T11:39:22.878 INFO:teuthology.orchestra.run.smithi012.stderr: 19750K .......... .......... .......... .......... .......... 21% 9.51M 14s 2023-12-03T11:39:22.880 INFO:teuthology.orchestra.run.smithi012.stderr: 19800K .......... .......... .......... .......... .......... 21% 17.8M 14s 2023-12-03T11:39:22.884 INFO:teuthology.orchestra.run.smithi012.stderr: 19850K .......... .......... .......... .......... .......... 21% 12.1M 14s 2023-12-03T11:39:22.890 INFO:teuthology.orchestra.run.smithi012.stderr: 19900K .......... .......... .......... .......... .......... 21% 8.38M 14s 2023-12-03T11:39:22.895 INFO:teuthology.orchestra.run.smithi012.stderr: 19950K .......... .......... .......... .......... .......... 21% 12.2M 14s 2023-12-03T11:39:22.900 INFO:teuthology.orchestra.run.smithi012.stderr: 20000K .......... .......... .......... .......... .......... 21% 9.34M 14s 2023-12-03T11:39:22.905 INFO:teuthology.orchestra.run.smithi012.stderr: 20050K .......... .......... .......... .......... .......... 21% 9.62M 14s 2023-12-03T11:39:22.909 INFO:teuthology.orchestra.run.smithi012.stderr: 20100K .......... .......... .......... .......... .......... 21% 12.3M 14s 2023-12-03T11:39:22.912 INFO:teuthology.orchestra.run.smithi012.stderr: 20150K .......... .......... .......... .......... .......... 21% 12.6M 14s 2023-12-03T11:39:22.917 INFO:teuthology.orchestra.run.smithi012.stderr: 20200K .......... .......... .......... .......... .......... 21% 9.94M 14s 2023-12-03T11:39:22.926 INFO:teuthology.orchestra.run.smithi012.stderr: 20250K .......... .......... .......... .......... .......... 21% 9.19M 14s 2023-12-03T11:39:22.926 INFO:teuthology.orchestra.run.smithi012.stderr: 20300K .......... .......... .......... .......... .......... 21% 19.2M 13s 2023-12-03T11:39:22.930 INFO:teuthology.orchestra.run.smithi012.stderr: 20350K .......... .......... .......... .......... .......... 21% 12.2M 13s 2023-12-03T11:39:22.936 INFO:teuthology.orchestra.run.smithi012.stderr: 20400K .......... .......... .......... .......... .......... 22% 6.59M 13s 2023-12-03T11:39:22.940 INFO:teuthology.orchestra.run.smithi012.stderr: 20450K .......... .......... .......... .......... .......... 22% 14.2M 13s 2023-12-03T11:39:22.944 INFO:teuthology.orchestra.run.smithi012.stderr: 20500K .......... .......... .......... .......... .......... 22% 12.5M 13s 2023-12-03T11:39:22.949 INFO:teuthology.orchestra.run.smithi012.stderr: 20550K .......... .......... .......... .......... .......... 22% 9.18M 13s 2023-12-03T11:39:22.953 INFO:teuthology.orchestra.run.smithi012.stderr: 20600K .......... .......... .......... .......... .......... 22% 12.8M 13s 2023-12-03T11:39:22.957 INFO:teuthology.orchestra.run.smithi012.stderr: 20650K .......... .......... .......... .......... .......... 22% 12.6M 13s 2023-12-03T11:39:22.962 INFO:teuthology.orchestra.run.smithi012.stderr: 20700K .......... .......... .......... .......... .......... 22% 9.65M 13s 2023-12-03T11:39:22.968 INFO:teuthology.orchestra.run.smithi012.stderr: 20750K .......... .......... .......... .......... .......... 22% 8.66M 13s 2023-12-03T11:39:22.971 INFO:teuthology.orchestra.run.smithi012.stderr: 20800K .......... .......... .......... .......... .......... 22% 15.1M 13s 2023-12-03T11:39:22.975 INFO:teuthology.orchestra.run.smithi012.stderr: 20850K .......... .......... .......... .......... .......... 22% 12.4M 13s 2023-12-03T11:39:22.981 INFO:teuthology.orchestra.run.smithi012.stderr: 20900K .......... .......... .......... .......... .......... 22% 7.91M 13s 2023-12-03T11:39:22.985 INFO:teuthology.orchestra.run.smithi012.stderr: 20950K .......... .......... .......... .......... .......... 22% 12.5M 13s 2023-12-03T11:39:22.988 INFO:teuthology.orchestra.run.smithi012.stderr: 21000K .......... .......... .......... .......... .......... 22% 13.6M 13s 2023-12-03T11:39:22.994 INFO:teuthology.orchestra.run.smithi012.stderr: 21050K .......... .......... .......... .......... .......... 22% 8.55M 13s 2023-12-03T11:39:22.997 INFO:teuthology.orchestra.run.smithi012.stderr: 21100K .......... .......... .......... .......... .......... 22% 13.6M 13s 2023-12-03T11:39:23.003 INFO:teuthology.orchestra.run.smithi012.stderr: 21150K .......... .......... .......... .......... .......... 22% 13.1M 13s 2023-12-03T11:39:23.009 INFO:teuthology.orchestra.run.smithi012.stderr: 21200K .......... .......... .......... .......... .......... 22% 6.62M 13s 2023-12-03T11:39:23.012 INFO:teuthology.orchestra.run.smithi012.stderr: 21250K .......... .......... .......... .......... .......... 22% 14.4M 13s 2023-12-03T11:39:23.015 INFO:teuthology.orchestra.run.smithi012.stderr: 21300K .......... .......... .......... .......... .......... 22% 14.7M 13s 2023-12-03T11:39:23.019 INFO:teuthology.orchestra.run.smithi012.stderr: 21350K .......... .......... .......... .......... .......... 23% 13.7M 13s 2023-12-03T11:39:23.025 INFO:teuthology.orchestra.run.smithi012.stderr: 21400K .......... .......... .......... .......... .......... 23% 7.74M 13s 2023-12-03T11:39:23.028 INFO:teuthology.orchestra.run.smithi012.stderr: 21450K .......... .......... .......... .......... .......... 23% 17.7M 13s 2023-12-03T11:39:23.031 INFO:teuthology.orchestra.run.smithi012.stderr: 21500K .......... .......... .......... .......... .......... 23% 13.6M 13s 2023-12-03T11:39:23.038 INFO:teuthology.orchestra.run.smithi012.stderr: 21550K .......... .......... .......... .......... .......... 23% 11.3M 13s 2023-12-03T11:39:23.042 INFO:teuthology.orchestra.run.smithi012.stderr: 21600K .......... .......... .......... .......... .......... 23% 7.87M 13s 2023-12-03T11:39:23.045 INFO:teuthology.orchestra.run.smithi012.stderr: 21650K .......... .......... .......... .......... .......... 23% 14.5M 13s 2023-12-03T11:39:23.049 INFO:teuthology.orchestra.run.smithi012.stderr: 21700K .......... .......... .......... .......... .......... 23% 14.0M 13s 2023-12-03T11:39:23.055 INFO:teuthology.orchestra.run.smithi012.stderr: 21750K .......... .......... .......... .......... .......... 23% 7.72M 13s 2023-12-03T11:39:23.059 INFO:teuthology.orchestra.run.smithi012.stderr: 21800K .......... .......... .......... .......... .......... 23% 13.8M 13s 2023-12-03T11:39:23.062 INFO:teuthology.orchestra.run.smithi012.stderr: 21850K .......... .......... .......... .......... .......... 23% 16.1M 13s 2023-12-03T11:39:23.066 INFO:teuthology.orchestra.run.smithi012.stderr: 21900K .......... .......... .......... .......... .......... 23% 12.4M 13s 2023-12-03T11:39:23.072 INFO:teuthology.orchestra.run.smithi012.stderr: 21950K .......... .......... .......... .......... .......... 23% 9.94M 13s 2023-12-03T11:39:23.076 INFO:teuthology.orchestra.run.smithi012.stderr: 22000K .......... .......... .......... .......... .......... 23% 9.44M 13s 2023-12-03T11:39:23.079 INFO:teuthology.orchestra.run.smithi012.stderr: 22050K .......... .......... .......... .......... .......... 23% 14.3M 13s 2023-12-03T11:39:23.085 INFO:teuthology.orchestra.run.smithi012.stderr: 22100K .......... .......... .......... .......... .......... 23% 8.55M 13s 2023-12-03T11:39:23.088 INFO:teuthology.orchestra.run.smithi012.stderr: 22150K .......... .......... .......... .......... .......... 23% 13.7M 13s 2023-12-03T11:39:23.092 INFO:teuthology.orchestra.run.smithi012.stderr: 22200K .......... .......... .......... .......... .......... 23% 14.7M 13s 2023-12-03T11:39:23.095 INFO:teuthology.orchestra.run.smithi012.stderr: 22250K .......... .......... .......... .......... .......... 24% 13.9M 13s 2023-12-03T11:39:23.101 INFO:teuthology.orchestra.run.smithi012.stderr: 22300K .......... .......... .......... .......... .......... 24% 9.39M 12s 2023-12-03T11:39:23.106 INFO:teuthology.orchestra.run.smithi012.stderr: 22350K .......... .......... .......... .......... .......... 24% 13.0M 12s 2023-12-03T11:39:23.109 INFO:teuthology.orchestra.run.smithi012.stderr: 22400K .......... .......... .......... .......... .......... 24% 10.7M 12s 2023-12-03T11:39:23.115 INFO:teuthology.orchestra.run.smithi012.stderr: 22450K .......... .......... .......... .......... .......... 24% 8.36M 12s 2023-12-03T11:39:23.119 INFO:teuthology.orchestra.run.smithi012.stderr: 22500K .......... .......... .......... .......... .......... 24% 12.0M 12s 2023-12-03T11:39:23.121 INFO:teuthology.orchestra.run.smithi012.stderr: 22550K .......... .......... .......... .......... .......... 24% 18.5M 12s 2023-12-03T11:39:23.125 INFO:teuthology.orchestra.run.smithi012.stderr: 22600K .......... .......... .......... .......... .......... 24% 13.1M 12s 2023-12-03T11:39:23.131 INFO:teuthology.orchestra.run.smithi012.stderr: 22650K .......... .......... .......... .......... .......... 24% 8.63M 12s 2023-12-03T11:39:23.135 INFO:teuthology.orchestra.run.smithi012.stderr: 22700K .......... .......... .......... .......... .......... 24% 12.3M 12s 2023-12-03T11:39:23.139 INFO:teuthology.orchestra.run.smithi012.stderr: 22750K .......... .......... .......... .......... .......... 24% 19.3M 12s 2023-12-03T11:39:23.145 INFO:teuthology.orchestra.run.smithi012.stderr: 22800K .......... .......... .......... .......... .......... 24% 6.62M 12s 2023-12-03T11:39:23.148 INFO:teuthology.orchestra.run.smithi012.stderr: 22850K .......... .......... .......... .......... .......... 24% 13.9M 12s 2023-12-03T11:39:23.151 INFO:teuthology.orchestra.run.smithi012.stderr: 22900K .......... .......... .......... .......... .......... 24% 16.4M 12s 2023-12-03T11:39:23.155 INFO:teuthology.orchestra.run.smithi012.stderr: 22950K .......... .......... .......... .......... .......... 24% 13.2M 12s 2023-12-03T11:39:23.160 INFO:teuthology.orchestra.run.smithi012.stderr: 23000K .......... .......... .......... .......... .......... 24% 8.76M 12s 2023-12-03T11:39:23.164 INFO:teuthology.orchestra.run.smithi012.stderr: 23050K .......... .......... .......... .......... .......... 24% 15.5M 12s 2023-12-03T11:39:23.167 INFO:teuthology.orchestra.run.smithi012.stderr: 23100K .......... .......... .......... .......... .......... 24% 14.6M 12s 2023-12-03T11:39:23.173 INFO:teuthology.orchestra.run.smithi012.stderr: 23150K .......... .......... .......... .......... .......... 24% 8.96M 12s 2023-12-03T11:39:23.178 INFO:teuthology.orchestra.run.smithi012.stderr: 23200K .......... .......... .......... .......... .......... 25% 9.11M 12s 2023-12-03T11:39:23.181 INFO:teuthology.orchestra.run.smithi012.stderr: 23250K .......... .......... .......... .......... .......... 25% 14.9M 12s 2023-12-03T11:39:23.184 INFO:teuthology.orchestra.run.smithi012.stderr: 23300K .......... .......... .......... .......... .......... 25% 14.3M 12s 2023-12-03T11:39:23.190 INFO:teuthology.orchestra.run.smithi012.stderr: 23350K .......... .......... .......... .......... .......... 25% 9.86M 12s 2023-12-03T11:39:23.193 INFO:teuthology.orchestra.run.smithi012.stderr: 23400K .......... .......... .......... .......... .......... 25% 12.1M 12s 2023-12-03T11:39:23.197 INFO:teuthology.orchestra.run.smithi012.stderr: 23450K .......... .......... .......... .......... .......... 25% 14.6M 12s 2023-12-03T11:39:23.202 INFO:teuthology.orchestra.run.smithi012.stderr: 23500K .......... .......... .......... .......... .......... 25% 8.96M 12s 2023-12-03T11:39:23.207 INFO:teuthology.orchestra.run.smithi012.stderr: 23550K .......... .......... .......... .......... .......... 25% 18.1M 12s 2023-12-03T11:39:23.211 INFO:teuthology.orchestra.run.smithi012.stderr: 23600K .......... .......... .......... .......... .......... 25% 8.62M 12s 2023-12-03T11:39:23.213 INFO:teuthology.orchestra.run.smithi012.stderr: 23650K .......... .......... .......... .......... .......... 25% 18.3M 12s 2023-12-03T11:39:23.219 INFO:teuthology.orchestra.run.smithi012.stderr: 23700K .......... .......... .......... .......... .......... 25% 8.50M 12s 2023-12-03T11:39:23.223 INFO:teuthology.orchestra.run.smithi012.stderr: 23750K .......... .......... .......... .......... .......... 25% 12.6M 12s 2023-12-03T11:39:23.226 INFO:teuthology.orchestra.run.smithi012.stderr: 23800K .......... .......... .......... .......... .......... 25% 14.5M 12s 2023-12-03T11:39:23.229 INFO:teuthology.orchestra.run.smithi012.stderr: 23850K .......... .......... .......... .......... .......... 25% 18.5M 12s 2023-12-03T11:39:23.235 INFO:teuthology.orchestra.run.smithi012.stderr: 23900K .......... .......... .......... .......... .......... 25% 8.03M 12s 2023-12-03T11:39:23.239 INFO:teuthology.orchestra.run.smithi012.stderr: 23950K .......... .......... .......... .......... .......... 25% 12.5M 12s 2023-12-03T11:39:23.243 INFO:teuthology.orchestra.run.smithi012.stderr: 24000K .......... .......... .......... .......... .......... 25% 12.6M 12s 2023-12-03T11:39:23.249 INFO:teuthology.orchestra.run.smithi012.stderr: 24050K .......... .......... .......... .......... .......... 25% 7.86M 12s 2023-12-03T11:39:23.253 INFO:teuthology.orchestra.run.smithi012.stderr: 24100K .......... .......... .......... .......... .......... 25% 12.8M 12s 2023-12-03T11:39:23.255 INFO:teuthology.orchestra.run.smithi012.stderr: 24150K .......... .......... .......... .......... .......... 26% 22.0M 12s 2023-12-03T11:39:23.259 INFO:teuthology.orchestra.run.smithi012.stderr: 24200K .......... .......... .......... .......... .......... 26% 12.5M 12s 2023-12-03T11:39:23.265 INFO:teuthology.orchestra.run.smithi012.stderr: 24250K .......... .......... .......... .......... .......... 26% 8.49M 12s 2023-12-03T11:39:23.270 INFO:teuthology.orchestra.run.smithi012.stderr: 24300K .......... .......... .......... .......... .......... 26% 9.75M 12s 2023-12-03T11:39:23.274 INFO:teuthology.orchestra.run.smithi012.stderr: 24350K .......... .......... .......... .......... .......... 26% 17.5M 12s 2023-12-03T11:39:23.278 INFO:teuthology.orchestra.run.smithi012.stderr: 24400K .......... .......... .......... .......... .......... 26% 8.68M 12s 2023-12-03T11:39:23.283 INFO:teuthology.orchestra.run.smithi012.stderr: 24450K .......... .......... .......... .......... .......... 26% 10.1M 12s 2023-12-03T11:39:23.287 INFO:teuthology.orchestra.run.smithi012.stderr: 24500K .......... .......... .......... .......... .......... 26% 12.9M 12s 2023-12-03T11:39:23.290 INFO:teuthology.orchestra.run.smithi012.stderr: 24550K .......... .......... .......... .......... .......... 26% 15.5M 12s 2023-12-03T11:39:23.293 INFO:teuthology.orchestra.run.smithi012.stderr: 24600K .......... .......... .......... .......... .......... 26% 13.7M 11s 2023-12-03T11:39:23.299 INFO:teuthology.orchestra.run.smithi012.stderr: 24650K .......... .......... .......... .......... .......... 26% 8.90M 11s 2023-12-03T11:39:23.302 INFO:teuthology.orchestra.run.smithi012.stderr: 24700K .......... .......... .......... .......... .......... 26% 14.6M 11s 2023-12-03T11:39:23.308 INFO:teuthology.orchestra.run.smithi012.stderr: 24750K .......... .......... .......... .......... .......... 26% 12.8M 11s 2023-12-03T11:39:23.312 INFO:teuthology.orchestra.run.smithi012.stderr: 24800K .......... .......... .......... .......... .......... 26% 7.99M 11s 2023-12-03T11:39:23.316 INFO:teuthology.orchestra.run.smithi012.stderr: 24850K .......... .......... .......... .......... .......... 26% 14.0M 11s 2023-12-03T11:39:23.319 INFO:teuthology.orchestra.run.smithi012.stderr: 24900K .......... .......... .......... .......... .......... 26% 14.5M 11s 2023-12-03T11:39:23.323 INFO:teuthology.orchestra.run.smithi012.stderr: 24950K .......... .......... .......... .......... .......... 26% 12.6M 11s 2023-12-03T11:39:23.328 INFO:teuthology.orchestra.run.smithi012.stderr: 25000K .......... .......... .......... .......... .......... 26% 8.92M 11s 2023-12-03T11:39:23.332 INFO:teuthology.orchestra.run.smithi012.stderr: 25050K .......... .......... .......... .......... .......... 27% 12.5M 11s 2023-12-03T11:39:23.337 INFO:teuthology.orchestra.run.smithi012.stderr: 25100K .......... .......... .......... .......... .......... 27% 22.6M 11s 2023-12-03T11:39:23.339 INFO:teuthology.orchestra.run.smithi012.stderr: 25150K .......... .......... .......... .......... .......... 27% 14.1M 11s 2023-12-03T11:39:23.344 INFO:teuthology.orchestra.run.smithi012.stderr: 25200K .......... .......... .......... .......... .......... 27% 7.97M 11s 2023-12-03T11:39:23.348 INFO:teuthology.orchestra.run.smithi012.stderr: 25250K .......... .......... .......... .......... .......... 27% 13.2M 11s 2023-12-03T11:39:23.351 INFO:teuthology.orchestra.run.smithi012.stderr: 25300K .......... .......... .......... .......... .......... 27% 14.8M 11s 2023-12-03T11:39:23.354 INFO:teuthology.orchestra.run.smithi012.stderr: 25350K .......... .......... .......... .......... .......... 27% 16.2M 11s 2023-12-03T11:39:23.360 INFO:teuthology.orchestra.run.smithi012.stderr: 25400K .......... .......... .......... .......... .......... 27% 8.53M 11s 2023-12-03T11:39:23.363 INFO:teuthology.orchestra.run.smithi012.stderr: 25450K .......... .......... .......... .......... .......... 27% 14.3M 11s 2023-12-03T11:39:23.366 INFO:teuthology.orchestra.run.smithi012.stderr: 25500K .......... .......... .......... .......... .......... 27% 16.8M 11s 2023-12-03T11:39:23.370 INFO:teuthology.orchestra.run.smithi012.stderr: 25550K .......... .......... .......... .......... .......... 27% 15.6M 11s 2023-12-03T11:39:23.377 INFO:teuthology.orchestra.run.smithi012.stderr: 25600K .......... .......... .......... .......... .......... 27% 6.15M 11s 2023-12-03T11:39:23.380 INFO:teuthology.orchestra.run.smithi012.stderr: 25650K .......... .......... .......... .......... .......... 27% 17.7M 11s 2023-12-03T11:39:23.383 INFO:teuthology.orchestra.run.smithi012.stderr: 25700K .......... .......... .......... .......... .......... 27% 16.1M 11s 2023-12-03T11:39:23.386 INFO:teuthology.orchestra.run.smithi012.stderr: 25750K .......... .......... .......... .......... .......... 27% 16.8M 11s 2023-12-03T11:39:23.392 INFO:teuthology.orchestra.run.smithi012.stderr: 25800K .......... .......... .......... .......... .......... 27% 7.79M 11s 2023-12-03T11:39:23.395 INFO:teuthology.orchestra.run.smithi012.stderr: 25850K .......... .......... .......... .......... .......... 27% 15.3M 11s 2023-12-03T11:39:23.398 INFO:teuthology.orchestra.run.smithi012.stderr: 25900K .......... .......... .......... .......... .......... 27% 16.0M 11s 2023-12-03T11:39:23.406 INFO:teuthology.orchestra.run.smithi012.stderr: 25950K .......... .......... .......... .......... .......... 27% 16.6M 11s 2023-12-03T11:39:23.409 INFO:teuthology.orchestra.run.smithi012.stderr: 26000K .......... .......... .......... .......... .......... 28% 6.55M 11s 2023-12-03T11:39:23.412 INFO:teuthology.orchestra.run.smithi012.stderr: 26050K .......... .......... .......... .......... .......... 28% 16.8M 11s 2023-12-03T11:39:23.414 INFO:teuthology.orchestra.run.smithi012.stderr: 26100K .......... .......... .......... .......... .......... 28% 16.8M 11s 2023-12-03T11:39:23.421 INFO:teuthology.orchestra.run.smithi012.stderr: 26150K .......... .......... .......... .......... .......... 28% 7.29M 11s 2023-12-03T11:39:23.425 INFO:teuthology.orchestra.run.smithi012.stderr: 26200K .......... .......... .......... .......... .......... 28% 14.5M 11s 2023-12-03T11:39:23.428 INFO:teuthology.orchestra.run.smithi012.stderr: 26250K .......... .......... .......... .......... .......... 28% 15.7M 11s 2023-12-03T11:39:23.430 INFO:teuthology.orchestra.run.smithi012.stderr: 26300K .......... .......... .......... .......... .......... 28% 23.0M 11s 2023-12-03T11:39:23.438 INFO:teuthology.orchestra.run.smithi012.stderr: 26350K .......... .......... .......... .......... .......... 28% 7.67M 11s 2023-12-03T11:39:23.440 INFO:teuthology.orchestra.run.smithi012.stderr: 26400K .......... .......... .......... .......... .......... 28% 13.9M 11s 2023-12-03T11:39:23.443 INFO:teuthology.orchestra.run.smithi012.stderr: 26450K .......... .......... .......... .......... .......... 28% 13.0M 11s 2023-12-03T11:39:23.446 INFO:teuthology.orchestra.run.smithi012.stderr: 26500K .......... .......... .......... .......... .......... 28% 16.0M 11s 2023-12-03T11:39:23.453 INFO:teuthology.orchestra.run.smithi012.stderr: 26550K .......... .......... .......... .......... .......... 28% 7.90M 11s 2023-12-03T11:39:23.455 INFO:teuthology.orchestra.run.smithi012.stderr: 26600K .......... .......... .......... .......... .......... 28% 17.9M 11s 2023-12-03T11:39:23.458 INFO:teuthology.orchestra.run.smithi012.stderr: 26650K .......... .......... .......... .......... .......... 28% 17.9M 11s 2023-12-03T11:39:23.462 INFO:teuthology.orchestra.run.smithi012.stderr: 26700K .......... .......... .......... .......... .......... 28% 13.9M 11s 2023-12-03T11:39:23.468 INFO:teuthology.orchestra.run.smithi012.stderr: 26750K .......... .......... .......... .......... .......... 28% 8.44M 11s 2023-12-03T11:39:23.472 INFO:teuthology.orchestra.run.smithi012.stderr: 26800K .......... .......... .......... .......... .......... 28% 12.2M 11s 2023-12-03T11:39:23.475 INFO:teuthology.orchestra.run.smithi012.stderr: 26850K .......... .......... .......... .......... .......... 28% 15.1M 11s 2023-12-03T11:39:23.481 INFO:teuthology.orchestra.run.smithi012.stderr: 26900K .......... .......... .......... .......... .......... 29% 8.23M 11s 2023-12-03T11:39:23.483 INFO:teuthology.orchestra.run.smithi012.stderr: 26950K .......... .......... .......... .......... .......... 29% 21.5M 11s 2023-12-03T11:39:23.486 INFO:teuthology.orchestra.run.smithi012.stderr: 27000K .......... .......... .......... .......... .......... 29% 13.6M 11s 2023-12-03T11:39:23.489 INFO:teuthology.orchestra.run.smithi012.stderr: 27050K .......... .......... .......... .......... .......... 29% 16.5M 11s 2023-12-03T11:39:23.496 INFO:teuthology.orchestra.run.smithi012.stderr: 27100K .......... .......... .......... .......... .......... 29% 7.81M 11s 2023-12-03T11:39:23.499 INFO:teuthology.orchestra.run.smithi012.stderr: 27150K .......... .......... .......... .......... .......... 29% 21.8M 11s 2023-12-03T11:39:23.502 INFO:teuthology.orchestra.run.smithi012.stderr: 27200K .......... .......... .......... .......... .......... 29% 12.4M 10s 2023-12-03T11:39:23.505 INFO:teuthology.orchestra.run.smithi012.stderr: 27250K .......... .......... .......... .......... .......... 29% 15.5M 10s 2023-12-03T11:39:23.511 INFO:teuthology.orchestra.run.smithi012.stderr: 27300K .......... .......... .......... .......... .......... 29% 8.15M 10s 2023-12-03T11:39:23.514 INFO:teuthology.orchestra.run.smithi012.stderr: 27350K .......... .......... .......... .......... .......... 29% 14.3M 10s 2023-12-03T11:39:23.517 INFO:teuthology.orchestra.run.smithi012.stderr: 27400K .......... .......... .......... .......... .......... 29% 19.8M 10s 2023-12-03T11:39:23.520 INFO:teuthology.orchestra.run.smithi012.stderr: 27450K .......... .......... .......... .......... .......... 29% 16.2M 10s 2023-12-03T11:39:23.526 INFO:teuthology.orchestra.run.smithi012.stderr: 27500K .......... .......... .......... .......... .......... 29% 7.98M 10s 2023-12-03T11:39:23.530 INFO:teuthology.orchestra.run.smithi012.stderr: 27550K .......... .......... .......... .......... .......... 29% 15.2M 10s 2023-12-03T11:39:23.533 INFO:teuthology.orchestra.run.smithi012.stderr: 27600K .......... .......... .......... .......... .......... 29% 12.9M 10s 2023-12-03T11:39:23.537 INFO:teuthology.orchestra.run.smithi012.stderr: 27650K .......... .......... .......... .......... .......... 29% 17.4M 10s 2023-12-03T11:39:23.542 INFO:teuthology.orchestra.run.smithi012.stderr: 27700K .......... .......... .......... .......... .......... 29% 8.05M 10s 2023-12-03T11:39:23.545 INFO:teuthology.orchestra.run.smithi012.stderr: 27750K .......... .......... .......... .......... .......... 29% 16.1M 10s 2023-12-03T11:39:23.548 INFO:teuthology.orchestra.run.smithi012.stderr: 27800K .......... .......... .......... .......... .......... 29% 15.6M 10s 2023-12-03T11:39:23.551 INFO:teuthology.orchestra.run.smithi012.stderr: 27850K .......... .......... .......... .......... .......... 30% 17.1M 10s 2023-12-03T11:39:23.557 INFO:teuthology.orchestra.run.smithi012.stderr: 27900K .......... .......... .......... .......... .......... 30% 8.14M 10s 2023-12-03T11:39:23.561 INFO:teuthology.orchestra.run.smithi012.stderr: 27950K .......... .......... .......... .......... .......... 30% 15.9M 10s 2023-12-03T11:39:23.564 INFO:teuthology.orchestra.run.smithi012.stderr: 28000K .......... .......... .......... .......... .......... 30% 12.0M 10s 2023-12-03T11:39:23.570 INFO:teuthology.orchestra.run.smithi012.stderr: 28050K .......... .......... .......... .......... .......... 30% 8.37M 10s 2023-12-03T11:39:23.572 INFO:teuthology.orchestra.run.smithi012.stderr: 28100K .......... .......... .......... .......... .......... 30% 18.5M 10s 2023-12-03T11:39:23.575 INFO:teuthology.orchestra.run.smithi012.stderr: 28150K .......... .......... .......... .......... .......... 30% 16.2M 10s 2023-12-03T11:39:23.579 INFO:teuthology.orchestra.run.smithi012.stderr: 28200K .......... .......... .......... .......... .......... 30% 14.8M 10s 2023-12-03T11:39:23.585 INFO:teuthology.orchestra.run.smithi012.stderr: 28250K .......... .......... .......... .......... .......... 30% 8.30M 10s 2023-12-03T11:39:23.588 INFO:teuthology.orchestra.run.smithi012.stderr: 28300K .......... .......... .......... .......... .......... 30% 16.7M 10s 2023-12-03T11:39:23.591 INFO:teuthology.orchestra.run.smithi012.stderr: 28350K .......... .......... .......... .......... .......... 30% 16.8M 10s 2023-12-03T11:39:23.594 INFO:teuthology.orchestra.run.smithi012.stderr: 28400K .......... .......... .......... .......... .......... 30% 12.1M 10s 2023-12-03T11:39:23.600 INFO:teuthology.orchestra.run.smithi012.stderr: 28450K .......... .......... .......... .......... .......... 30% 8.83M 10s 2023-12-03T11:39:23.604 INFO:teuthology.orchestra.run.smithi012.stderr: 28500K .......... .......... .......... .......... .......... 30% 12.7M 10s 2023-12-03T11:39:23.606 INFO:teuthology.orchestra.run.smithi012.stderr: 28550K .......... .......... .......... .......... .......... 30% 20.6M 10s 2023-12-03T11:39:23.609 INFO:teuthology.orchestra.run.smithi012.stderr: 28600K .......... .......... .......... .......... .......... 30% 15.6M 10s 2023-12-03T11:39:23.615 INFO:teuthology.orchestra.run.smithi012.stderr: 28650K .......... .......... .......... .......... .......... 30% 8.79M 10s 2023-12-03T11:39:23.619 INFO:teuthology.orchestra.run.smithi012.stderr: 28700K .......... .......... .......... .......... .......... 30% 13.1M 10s 2023-12-03T11:39:23.622 INFO:teuthology.orchestra.run.smithi012.stderr: 28750K .......... .......... .......... .......... .......... 31% 15.3M 10s 2023-12-03T11:39:23.626 INFO:teuthology.orchestra.run.smithi012.stderr: 28800K .......... .......... .......... .......... .......... 31% 12.6M 10s 2023-12-03T11:39:23.632 INFO:teuthology.orchestra.run.smithi012.stderr: 28850K .......... .......... .......... .......... .......... 31% 7.49M 10s 2023-12-03T11:39:23.635 INFO:teuthology.orchestra.run.smithi012.stderr: 28900K .......... .......... .......... .......... .......... 31% 17.1M 10s 2023-12-03T11:39:23.637 INFO:teuthology.orchestra.run.smithi012.stderr: 28950K .......... .......... .......... .......... .......... 31% 26.4M 10s 2023-12-03T11:39:23.641 INFO:teuthology.orchestra.run.smithi012.stderr: 29000K .......... .......... .......... .......... .......... 31% 13.3M 10s 2023-12-03T11:39:23.646 INFO:teuthology.orchestra.run.smithi012.stderr: 29050K .......... .......... .......... .......... .......... 31% 8.17M 10s 2023-12-03T11:39:23.651 INFO:teuthology.orchestra.run.smithi012.stderr: 29100K .......... .......... .......... .......... .......... 31% 10.6M 10s 2023-12-03T11:39:23.655 INFO:teuthology.orchestra.run.smithi012.stderr: 29150K .......... .......... .......... .......... .......... 31% 16.2M 10s 2023-12-03T11:39:23.659 INFO:teuthology.orchestra.run.smithi012.stderr: 29200K .......... .......... .......... .......... .......... 31% 10.0M 10s 2023-12-03T11:39:23.662 INFO:teuthology.orchestra.run.smithi012.stderr: 29250K .......... .......... .......... .......... .......... 31% 19.5M 10s 2023-12-03T11:39:23.666 INFO:teuthology.orchestra.run.smithi012.stderr: 29300K .......... .......... .......... .......... .......... 31% 10.4M 10s 2023-12-03T11:39:23.670 INFO:teuthology.orchestra.run.smithi012.stderr: 29350K .......... .......... .......... .......... .......... 31% 12.4M 10s 2023-12-03T11:39:23.673 INFO:teuthology.orchestra.run.smithi012.stderr: 29400K .......... .......... .......... .......... .......... 31% 16.3M 10s 2023-12-03T11:39:23.677 INFO:teuthology.orchestra.run.smithi012.stderr: 29450K .......... .......... .......... .......... .......... 31% 15.4M 10s 2023-12-03T11:39:23.681 INFO:teuthology.orchestra.run.smithi012.stderr: 29500K .......... .......... .......... .......... .......... 31% 9.79M 10s 2023-12-03T11:39:23.686 INFO:teuthology.orchestra.run.smithi012.stderr: 29550K .......... .......... .......... .......... .......... 31% 15.7M 10s 2023-12-03T11:39:23.688 INFO:teuthology.orchestra.run.smithi012.stderr: 29600K .......... .......... .......... .......... .......... 31% 12.8M 10s 2023-12-03T11:39:23.691 INFO:teuthology.orchestra.run.smithi012.stderr: 29650K .......... .......... .......... .......... .......... 31% 15.3M 10s 2023-12-03T11:39:23.697 INFO:teuthology.orchestra.run.smithi012.stderr: 29700K .......... .......... .......... .......... .......... 32% 9.53M 10s 2023-12-03T11:39:23.700 INFO:teuthology.orchestra.run.smithi012.stderr: 29750K .......... .......... .......... .......... .......... 32% 16.8M 10s 2023-12-03T11:39:23.703 INFO:teuthology.orchestra.run.smithi012.stderr: 29800K .......... .......... .......... .......... .......... 32% 14.2M 10s 2023-12-03T11:39:23.706 INFO:teuthology.orchestra.run.smithi012.stderr: 29850K .......... .......... .......... .......... .......... 32% 17.2M 10s 2023-12-03T11:39:23.711 INFO:teuthology.orchestra.run.smithi012.stderr: 29900K .......... .......... .......... .......... .......... 32% 9.46M 10s 2023-12-03T11:39:23.714 INFO:teuthology.orchestra.run.smithi012.stderr: 29950K .......... .......... .......... .......... .......... 32% 17.1M 10s 2023-12-03T11:39:23.718 INFO:teuthology.orchestra.run.smithi012.stderr: 30000K .......... .......... .......... .......... .......... 32% 11.6M 10s 2023-12-03T11:39:23.721 INFO:teuthology.orchestra.run.smithi012.stderr: 30050K .......... .......... .......... .......... .......... 32% 18.4M 10s 2023-12-03T11:39:23.726 INFO:teuthology.orchestra.run.smithi012.stderr: 30100K .......... .......... .......... .......... .......... 32% 9.45M 10s 2023-12-03T11:39:23.729 INFO:teuthology.orchestra.run.smithi012.stderr: 30150K .......... .......... .......... .......... .......... 32% 16.4M 10s 2023-12-03T11:39:23.732 INFO:teuthology.orchestra.run.smithi012.stderr: 30200K .......... .......... .......... .......... .......... 32% 15.9M 9s 2023-12-03T11:39:23.735 INFO:teuthology.orchestra.run.smithi012.stderr: 30250K .......... .......... .......... .......... .......... 32% 14.6M 9s 2023-12-03T11:39:23.741 INFO:teuthology.orchestra.run.smithi012.stderr: 30300K .......... .......... .......... .......... .......... 32% 14.0M 9s 2023-12-03T11:39:23.744 INFO:teuthology.orchestra.run.smithi012.stderr: 30350K .......... .......... .......... .......... .......... 32% 10.6M 9s 2023-12-03T11:39:23.747 INFO:teuthology.orchestra.run.smithi012.stderr: 30400K .......... .......... .......... .......... .......... 32% 14.0M 9s 2023-12-03T11:39:23.750 INFO:teuthology.orchestra.run.smithi012.stderr: 30450K .......... .......... .......... .......... .......... 32% 15.5M 9s 2023-12-03T11:39:23.754 INFO:teuthology.orchestra.run.smithi012.stderr: 30500K .......... .......... .......... .......... .......... 32% 13.4M 9s 2023-12-03T11:39:23.758 INFO:teuthology.orchestra.run.smithi012.stderr: 30550K .......... .......... .......... .......... .......... 32% 10.8M 9s 2023-12-03T11:39:23.761 INFO:teuthology.orchestra.run.smithi012.stderr: 30600K .......... .......... .......... .......... .......... 32% 20.2M 9s 2023-12-03T11:39:23.764 INFO:teuthology.orchestra.run.smithi012.stderr: 30650K .......... .......... .......... .......... .......... 33% 13.0M 9s 2023-12-03T11:39:23.767 INFO:teuthology.orchestra.run.smithi012.stderr: 30700K .......... .......... .......... .......... .......... 33% 20.5M 9s 2023-12-03T11:39:23.773 INFO:teuthology.orchestra.run.smithi012.stderr: 30750K .......... .......... .......... .......... .......... 33% 7.66M 9s 2023-12-03T11:39:23.776 INFO:teuthology.orchestra.run.smithi012.stderr: 30800K .......... .......... .......... .......... .......... 33% 17.7M 9s 2023-12-03T11:39:23.779 INFO:teuthology.orchestra.run.smithi012.stderr: 30850K .......... .......... .......... .......... .......... 33% 15.2M 9s 2023-12-03T11:39:23.782 INFO:teuthology.orchestra.run.smithi012.stderr: 30900K .......... .......... .......... .......... .......... 33% 19.4M 9s 2023-12-03T11:39:23.788 INFO:teuthology.orchestra.run.smithi012.stderr: 30950K .......... .......... .......... .......... .......... 33% 7.94M 9s 2023-12-03T11:39:23.790 INFO:teuthology.orchestra.run.smithi012.stderr: 31000K .......... .......... .......... .......... .......... 33% 19.1M 9s 2023-12-03T11:39:23.793 INFO:teuthology.orchestra.run.smithi012.stderr: 31050K .......... .......... .......... .......... .......... 33% 17.4M 9s 2023-12-03T11:39:23.796 INFO:teuthology.orchestra.run.smithi012.stderr: 31100K .......... .......... .......... .......... .......... 33% 16.8M 9s 2023-12-03T11:39:23.803 INFO:teuthology.orchestra.run.smithi012.stderr: 31150K .......... .......... .......... .......... .......... 33% 12.1M 9s 2023-12-03T11:39:23.806 INFO:teuthology.orchestra.run.smithi012.stderr: 31200K .......... .......... .......... .......... .......... 33% 9.36M 9s 2023-12-03T11:39:23.808 INFO:teuthology.orchestra.run.smithi012.stderr: 31250K .......... .......... .......... .......... .......... 33% 18.0M 9s 2023-12-03T11:39:23.811 INFO:teuthology.orchestra.run.smithi012.stderr: 31300K .......... .......... .......... .......... .......... 33% 16.6M 9s 2023-12-03T11:39:23.815 INFO:teuthology.orchestra.run.smithi012.stderr: 31350K .......... .......... .......... .......... .......... 33% 13.1M 9s 2023-12-03T11:39:23.819 INFO:teuthology.orchestra.run.smithi012.stderr: 31400K .......... .......... .......... .......... .......... 33% 10.4M 9s 2023-12-03T11:39:23.822 INFO:teuthology.orchestra.run.smithi012.stderr: 31450K .......... .......... .......... .......... .......... 33% 17.5M 9s 2023-12-03T11:39:23.825 INFO:teuthology.orchestra.run.smithi012.stderr: 31500K .......... .......... .......... .......... .......... 33% 19.8M 9s 2023-12-03T11:39:23.830 INFO:teuthology.orchestra.run.smithi012.stderr: 31550K .......... .......... .......... .......... .......... 34% 15.2M 9s 2023-12-03T11:39:23.834 INFO:teuthology.orchestra.run.smithi012.stderr: 31600K .......... .......... .......... .......... .......... 34% 7.87M 9s 2023-12-03T11:39:23.837 INFO:teuthology.orchestra.run.smithi012.stderr: 31650K .......... .......... .......... .......... .......... 34% 16.2M 9s 2023-12-03T11:39:23.839 INFO:teuthology.orchestra.run.smithi012.stderr: 31700K .......... .......... .......... .......... .......... 34% 19.9M 9s 2023-12-03T11:39:23.842 INFO:teuthology.orchestra.run.smithi012.stderr: 31750K .......... .......... .......... .......... .......... 34% 16.0M 9s 2023-12-03T11:39:23.848 INFO:teuthology.orchestra.run.smithi012.stderr: 31800K .......... .......... .......... .......... .......... 34% 8.62M 9s 2023-12-03T11:39:23.851 INFO:teuthology.orchestra.run.smithi012.stderr: 31850K .......... .......... .......... .......... .......... 34% 19.8M 9s 2023-12-03T11:39:23.854 INFO:teuthology.orchestra.run.smithi012.stderr: 31900K .......... .......... .......... .......... .......... 34% 16.8M 9s 2023-12-03T11:39:23.857 INFO:teuthology.orchestra.run.smithi012.stderr: 31950K .......... .......... .......... .......... .......... 34% 19.4M 9s 2023-12-03T11:39:23.863 INFO:teuthology.orchestra.run.smithi012.stderr: 32000K .......... .......... .......... .......... .......... 34% 7.21M 9s 2023-12-03T11:39:23.867 INFO:teuthology.orchestra.run.smithi012.stderr: 32050K .......... .......... .......... .......... .......... 34% 12.9M 9s 2023-12-03T11:39:23.868 INFO:teuthology.orchestra.run.smithi012.stderr: 32100K .......... .......... .......... .......... .......... 34% 28.4M 9s 2023-12-03T11:39:23.871 INFO:teuthology.orchestra.run.smithi012.stderr: 32150K .......... .......... .......... .......... .......... 34% 19.7M 9s 2023-12-03T11:39:23.878 INFO:teuthology.orchestra.run.smithi012.stderr: 32200K .......... .......... .......... .......... .......... 34% 7.27M 9s 2023-12-03T11:39:23.880 INFO:teuthology.orchestra.run.smithi012.stderr: 32250K .......... .......... .......... .......... .......... 34% 18.0M 9s 2023-12-03T11:39:23.883 INFO:teuthology.orchestra.run.smithi012.stderr: 32300K .......... .......... .......... .......... .......... 34% 15.6M 9s 2023-12-03T11:39:23.887 INFO:teuthology.orchestra.run.smithi012.stderr: 32350K .......... .......... .......... .......... .......... 34% 21.5M 9s 2023-12-03T11:39:23.895 INFO:teuthology.orchestra.run.smithi012.stderr: 32400K .......... .......... .......... .......... .......... 34% 7.05M 9s 2023-12-03T11:39:23.895 INFO:teuthology.orchestra.run.smithi012.stderr: 32450K .......... .......... .......... .......... .......... 34% 18.4M 9s 2023-12-03T11:39:23.898 INFO:teuthology.orchestra.run.smithi012.stderr: 32500K .......... .......... .......... .......... .......... 35% 17.6M 9s 2023-12-03T11:39:23.901 INFO:teuthology.orchestra.run.smithi012.stderr: 32550K .......... .......... .......... .......... .......... 35% 18.6M 9s 2023-12-03T11:39:23.907 INFO:teuthology.orchestra.run.smithi012.stderr: 32600K .......... .......... .......... .......... .......... 35% 7.98M 9s 2023-12-03T11:39:23.909 INFO:teuthology.orchestra.run.smithi012.stderr: 32650K .......... .......... .......... .......... .......... 35% 20.7M 9s 2023-12-03T11:39:23.912 INFO:teuthology.orchestra.run.smithi012.stderr: 32700K .......... .......... .......... .......... .......... 35% 18.2M 9s 2023-12-03T11:39:23.916 INFO:teuthology.orchestra.run.smithi012.stderr: 32750K .......... .......... .......... .......... .......... 35% 16.5M 9s 2023-12-03T11:39:23.922 INFO:teuthology.orchestra.run.smithi012.stderr: 32800K .......... .......... .......... .......... .......... 35% 7.26M 9s 2023-12-03T11:39:23.924 INFO:teuthology.orchestra.run.smithi012.stderr: 32850K .......... .......... .......... .......... .......... 35% 21.3M 9s 2023-12-03T11:39:23.927 INFO:teuthology.orchestra.run.smithi012.stderr: 32900K .......... .......... .......... .......... .......... 35% 17.0M 9s 2023-12-03T11:39:23.929 INFO:teuthology.orchestra.run.smithi012.stderr: 32950K .......... .......... .......... .......... .......... 35% 17.2M 9s 2023-12-03T11:39:23.933 INFO:teuthology.orchestra.run.smithi012.stderr: 33000K .......... .......... .......... .......... .......... 35% 14.1M 9s 2023-12-03T11:39:23.938 INFO:teuthology.orchestra.run.smithi012.stderr: 33050K .......... .......... .......... .......... .......... 35% 10.1M 9s 2023-12-03T11:39:23.941 INFO:teuthology.orchestra.run.smithi012.stderr: 33100K .......... .......... .......... .......... .......... 35% 15.9M 9s 2023-12-03T11:39:23.945 INFO:teuthology.orchestra.run.smithi012.stderr: 33150K .......... .......... .......... .......... .......... 35% 18.7M 9s 2023-12-03T11:39:23.948 INFO:teuthology.orchestra.run.smithi012.stderr: 33200K .......... .......... .......... .......... .......... 35% 11.6M 9s 2023-12-03T11:39:23.952 INFO:teuthology.orchestra.run.smithi012.stderr: 33250K .......... .......... .......... .......... .......... 35% 10.3M 9s 2023-12-03T11:39:23.956 INFO:teuthology.orchestra.run.smithi012.stderr: 33300K .......... .......... .......... .......... .......... 35% 15.8M 9s 2023-12-03T11:39:23.958 INFO:teuthology.orchestra.run.smithi012.stderr: 33350K .......... .......... .......... .......... .......... 35% 18.8M 9s 2023-12-03T11:39:23.961 INFO:teuthology.orchestra.run.smithi012.stderr: 33400K .......... .......... .......... .......... .......... 36% 17.8M 9s 2023-12-03T11:39:23.966 INFO:teuthology.orchestra.run.smithi012.stderr: 33450K .......... .......... .......... .......... .......... 36% 8.65M 9s 2023-12-03T11:39:23.970 INFO:teuthology.orchestra.run.smithi012.stderr: 33500K .......... .......... .......... .......... .......... 36% 14.7M 9s 2023-12-03T11:39:23.973 INFO:teuthology.orchestra.run.smithi012.stderr: 33550K .......... .......... .......... .......... .......... 36% 21.8M 9s 2023-12-03T11:39:23.976 INFO:teuthology.orchestra.run.smithi012.stderr: 33600K .......... .......... .......... .......... .......... 36% 13.3M 9s 2023-12-03T11:39:23.981 INFO:teuthology.orchestra.run.smithi012.stderr: 33650K .......... .......... .......... .......... .......... 36% 8.85M 8s 2023-12-03T11:39:23.984 INFO:teuthology.orchestra.run.smithi012.stderr: 33700K .......... .......... .......... .......... .......... 36% 16.8M 8s 2023-12-03T11:39:23.987 INFO:teuthology.orchestra.run.smithi012.stderr: 33750K .......... .......... .......... .......... .......... 36% 18.7M 8s 2023-12-03T11:39:23.990 INFO:teuthology.orchestra.run.smithi012.stderr: 33800K .......... .......... .......... .......... .......... 36% 17.0M 8s 2023-12-03T11:39:23.993 INFO:teuthology.orchestra.run.smithi012.stderr: 33850K .......... .......... .......... .......... .......... 36% 15.9M 8s 2023-12-03T11:39:23.998 INFO:teuthology.orchestra.run.smithi012.stderr: 33900K .......... .......... .......... .......... .......... 36% 9.74M 8s 2023-12-03T11:39:24.002 INFO:teuthology.orchestra.run.smithi012.stderr: 33950K .......... .......... .......... .......... .......... 36% 13.5M 8s 2023-12-03T11:39:24.005 INFO:teuthology.orchestra.run.smithi012.stderr: 34000K .......... .......... .......... .......... .......... 36% 14.9M 8s 2023-12-03T11:39:24.008 INFO:teuthology.orchestra.run.smithi012.stderr: 34050K .......... .......... .......... .......... .......... 36% 17.0M 8s 2023-12-03T11:39:24.012 INFO:teuthology.orchestra.run.smithi012.stderr: 34100K .......... .......... .......... .......... .......... 36% 10.5M 8s 2023-12-03T11:39:24.015 INFO:teuthology.orchestra.run.smithi012.stderr: 34150K .......... .......... .......... .......... .......... 36% 19.7M 8s 2023-12-03T11:39:24.019 INFO:teuthology.orchestra.run.smithi012.stderr: 34200K .......... .......... .......... .......... .......... 36% 10.7M 8s 2023-12-03T11:39:24.021 INFO:teuthology.orchestra.run.smithi012.stderr: 34250K .......... .......... .......... .......... .......... 36% 22.6M 8s 2023-12-03T11:39:24.026 INFO:teuthology.orchestra.run.smithi012.stderr: 34300K .......... .......... .......... .......... .......... 36% 10.1M 8s 2023-12-03T11:39:24.029 INFO:teuthology.orchestra.run.smithi012.stderr: 34350K .......... .......... .......... .......... .......... 37% 29.3M 8s 2023-12-03T11:39:24.032 INFO:teuthology.orchestra.run.smithi012.stderr: 34400K .......... .......... .......... .......... .......... 37% 10.9M 8s 2023-12-03T11:39:24.035 INFO:teuthology.orchestra.run.smithi012.stderr: 34450K .......... .......... .......... .......... .......... 37% 16.4M 8s 2023-12-03T11:39:24.040 INFO:teuthology.orchestra.run.smithi012.stderr: 34500K .......... .......... .......... .......... .......... 37% 9.70M 8s 2023-12-03T11:39:24.046 INFO:teuthology.orchestra.run.smithi012.stderr: 34550K .......... .......... .......... .......... .......... 37% 9.42M 8s 2023-12-03T11:39:24.048 INFO:teuthology.orchestra.run.smithi012.stderr: 34600K .......... .......... .......... .......... .......... 37% 20.9M 8s 2023-12-03T11:39:24.051 INFO:teuthology.orchestra.run.smithi012.stderr: 34650K .......... .......... .......... .......... .......... 37% 17.5M 8s 2023-12-03T11:39:24.053 INFO:teuthology.orchestra.run.smithi012.stderr: 34700K .......... .......... .......... .......... .......... 37% 18.1M 8s 2023-12-03T11:39:24.060 INFO:teuthology.orchestra.run.smithi012.stderr: 34750K .......... .......... .......... .......... .......... 37% 17.3M 8s 2023-12-03T11:39:24.063 INFO:teuthology.orchestra.run.smithi012.stderr: 34800K .......... .......... .......... .......... .......... 37% 7.78M 8s 2023-12-03T11:39:24.067 INFO:teuthology.orchestra.run.smithi012.stderr: 34850K .......... .......... .......... .......... .......... 37% 10.7M 8s 2023-12-03T11:39:24.069 INFO:teuthology.orchestra.run.smithi012.stderr: 34900K .......... .......... .......... .......... .......... 37% 22.4M 8s 2023-12-03T11:39:24.072 INFO:teuthology.orchestra.run.smithi012.stderr: 34950K .......... .......... .......... .......... .......... 37% 18.1M 8s 2023-12-03T11:39:24.077 INFO:teuthology.orchestra.run.smithi012.stderr: 35000K .......... .......... .......... .......... .......... 37% 11.0M 8s 2023-12-03T11:39:24.081 INFO:teuthology.orchestra.run.smithi012.stderr: 35050K .......... .......... .......... .......... .......... 37% 11.0M 8s 2023-12-03T11:39:24.083 INFO:teuthology.orchestra.run.smithi012.stderr: 35100K .......... .......... .......... .......... .......... 37% 23.8M 8s 2023-12-03T11:39:24.086 INFO:teuthology.orchestra.run.smithi012.stderr: 35150K .......... .......... .......... .......... .......... 37% 24.0M 8s 2023-12-03T11:39:24.092 INFO:teuthology.orchestra.run.smithi012.stderr: 35200K .......... .......... .......... .......... .......... 37% 7.64M 8s 2023-12-03T11:39:24.094 INFO:teuthology.orchestra.run.smithi012.stderr: 35250K .......... .......... .......... .......... .......... 37% 17.1M 8s 2023-12-03T11:39:24.098 INFO:teuthology.orchestra.run.smithi012.stderr: 35300K .......... .......... .......... .......... .......... 38% 14.5M 8s 2023-12-03T11:39:24.100 INFO:teuthology.orchestra.run.smithi012.stderr: 35350K .......... .......... .......... .......... .......... 38% 25.3M 8s 2023-12-03T11:39:24.105 INFO:teuthology.orchestra.run.smithi012.stderr: 35400K .......... .......... .......... .......... .......... 38% 8.99M 8s 2023-12-03T11:39:24.108 INFO:teuthology.orchestra.run.smithi012.stderr: 35450K .......... .......... .......... .......... .......... 38% 17.7M 8s 2023-12-03T11:39:24.111 INFO:teuthology.orchestra.run.smithi012.stderr: 35500K .......... .......... .......... .......... .......... 38% 15.6M 8s 2023-12-03T11:39:24.115 INFO:teuthology.orchestra.run.smithi012.stderr: 35550K .......... .......... .......... .......... .......... 38% 23.8M 8s 2023-12-03T11:39:24.117 INFO:teuthology.orchestra.run.smithi012.stderr: 35600K .......... .......... .......... .......... .......... 38% 12.5M 8s 2023-12-03T11:39:24.122 INFO:teuthology.orchestra.run.smithi012.stderr: 35650K .......... .......... .......... .......... .......... 38% 9.83M 8s 2023-12-03T11:39:24.124 INFO:teuthology.orchestra.run.smithi012.stderr: 35700K .......... .......... .......... .......... .......... 38% 19.1M 8s 2023-12-03T11:39:24.127 INFO:teuthology.orchestra.run.smithi012.stderr: 35750K .......... .......... .......... .......... .......... 38% 15.5M 8s 2023-12-03T11:39:24.131 INFO:teuthology.orchestra.run.smithi012.stderr: 35800K .......... .......... .......... .......... .......... 38% 19.2M 8s 2023-12-03T11:39:24.135 INFO:teuthology.orchestra.run.smithi012.stderr: 35850K .......... .......... .......... .......... .......... 38% 10.8M 8s 2023-12-03T11:39:24.138 INFO:teuthology.orchestra.run.smithi012.stderr: 35900K .......... .......... .......... .......... .......... 38% 16.3M 8s 2023-12-03T11:39:24.141 INFO:teuthology.orchestra.run.smithi012.stderr: 35950K .......... .......... .......... .......... .......... 38% 16.9M 8s 2023-12-03T11:39:24.145 INFO:teuthology.orchestra.run.smithi012.stderr: 36000K .......... .......... .......... .......... .......... 38% 12.9M 8s 2023-12-03T11:39:24.147 INFO:teuthology.orchestra.run.smithi012.stderr: 36050K .......... .......... .......... .......... .......... 38% 18.5M 8s 2023-12-03T11:39:24.154 INFO:teuthology.orchestra.run.smithi012.stderr: 36100K .......... .......... .......... .......... .......... 38% 10.0M 8s 2023-12-03T11:39:24.154 INFO:teuthology.orchestra.run.smithi012.stderr: 36150K .......... .......... .......... .......... .......... 38% 18.5M 8s 2023-12-03T11:39:24.157 INFO:teuthology.orchestra.run.smithi012.stderr: 36200K .......... .......... .......... .......... .......... 39% 16.2M 8s 2023-12-03T11:39:24.160 INFO:teuthology.orchestra.run.smithi012.stderr: 36250K .......... .......... .......... .......... .......... 39% 21.5M 8s 2023-12-03T11:39:24.164 INFO:teuthology.orchestra.run.smithi012.stderr: 36300K .......... .......... .......... .......... .......... 39% 11.0M 8s 2023-12-03T11:39:24.168 INFO:teuthology.orchestra.run.smithi012.stderr: 36350K .......... .......... .......... .......... .......... 39% 15.6M 8s 2023-12-03T11:39:24.171 INFO:teuthology.orchestra.run.smithi012.stderr: 36400K .......... .......... .......... .......... .......... 39% 13.2M 8s 2023-12-03T11:39:24.174 INFO:teuthology.orchestra.run.smithi012.stderr: 36450K .......... .......... .......... .......... .......... 39% 15.2M 8s 2023-12-03T11:39:24.176 INFO:teuthology.orchestra.run.smithi012.stderr: 36500K .......... .......... .......... .......... .......... 39% 20.3M 8s 2023-12-03T11:39:24.182 INFO:teuthology.orchestra.run.smithi012.stderr: 36550K .......... .......... .......... .......... .......... 39% 10.0M 8s 2023-12-03T11:39:24.184 INFO:teuthology.orchestra.run.smithi012.stderr: 36600K .......... .......... .......... .......... .......... 39% 18.9M 8s 2023-12-03T11:39:24.187 INFO:teuthology.orchestra.run.smithi012.stderr: 36650K .......... .......... .......... .......... .......... 39% 15.4M 8s 2023-12-03T11:39:24.189 INFO:teuthology.orchestra.run.smithi012.stderr: 36700K .......... .......... .......... .......... .......... 39% 20.9M 8s 2023-12-03T11:39:24.195 INFO:teuthology.orchestra.run.smithi012.stderr: 36750K .......... .......... .......... .......... .......... 39% 12.0M 8s 2023-12-03T11:39:24.198 INFO:teuthology.orchestra.run.smithi012.stderr: 36800K .......... .......... .......... .......... .......... 39% 10.6M 8s 2023-12-03T11:39:24.201 INFO:teuthology.orchestra.run.smithi012.stderr: 36850K .......... .......... .......... .......... .......... 39% 18.6M 8s 2023-12-03T11:39:24.204 INFO:teuthology.orchestra.run.smithi012.stderr: 36900K .......... .......... .......... .......... .......... 39% 15.2M 8s 2023-12-03T11:39:24.206 INFO:teuthology.orchestra.run.smithi012.stderr: 36950K .......... .......... .......... .......... .......... 39% 20.1M 8s 2023-12-03T11:39:24.211 INFO:teuthology.orchestra.run.smithi012.stderr: 37000K .......... .......... .......... .......... .......... 39% 10.2M 8s 2023-12-03T11:39:24.214 INFO:teuthology.orchestra.run.smithi012.stderr: 37050K .......... .......... .......... .......... .......... 39% 20.6M 8s 2023-12-03T11:39:24.217 INFO:teuthology.orchestra.run.smithi012.stderr: 37100K .......... .......... .......... .......... .......... 39% 14.0M 8s 2023-12-03T11:39:24.221 INFO:teuthology.orchestra.run.smithi012.stderr: 37150K .......... .......... .......... .......... .......... 40% 20.6M 8s 2023-12-03T11:39:24.225 INFO:teuthology.orchestra.run.smithi012.stderr: 37200K .......... .......... .......... .......... .......... 40% 8.61M 8s 2023-12-03T11:39:24.228 INFO:teuthology.orchestra.run.smithi012.stderr: 37250K .......... .......... .......... .......... .......... 40% 17.1M 8s 2023-12-03T11:39:24.230 INFO:teuthology.orchestra.run.smithi012.stderr: 37300K .......... .......... .......... .......... .......... 40% 19.5M 8s 2023-12-03T11:39:24.234 INFO:teuthology.orchestra.run.smithi012.stderr: 37350K .......... .......... .......... .......... .......... 40% 13.9M 8s 2023-12-03T11:39:24.236 INFO:teuthology.orchestra.run.smithi012.stderr: 37400K .......... .......... .......... .......... .......... 40% 20.4M 8s 2023-12-03T11:39:24.241 INFO:teuthology.orchestra.run.smithi012.stderr: 37450K .......... .......... .......... .......... .......... 40% 10.9M 8s 2023-12-03T11:39:24.243 INFO:teuthology.orchestra.run.smithi012.stderr: 37500K .......... .......... .......... .......... .......... 40% 18.6M 8s 2023-12-03T11:39:24.247 INFO:teuthology.orchestra.run.smithi012.stderr: 37550K .......... .......... .......... .......... .......... 40% 15.2M 8s 2023-12-03T11:39:24.251 INFO:teuthology.orchestra.run.smithi012.stderr: 37600K .......... .......... .......... .......... .......... 40% 12.3M 7s 2023-12-03T11:39:24.256 INFO:teuthology.orchestra.run.smithi012.stderr: 37650K .......... .......... .......... .......... .......... 40% 9.93M 7s 2023-12-03T11:39:24.257 INFO:teuthology.orchestra.run.smithi012.stderr: 37700K .......... .......... .......... .......... .......... 40% 25.2M 7s 2023-12-03T11:39:24.260 INFO:teuthology.orchestra.run.smithi012.stderr: 37750K .......... .......... .......... .......... .......... 40% 17.5M 7s 2023-12-03T11:39:24.264 INFO:teuthology.orchestra.run.smithi012.stderr: 37800K .......... .......... .......... .......... .......... 40% 13.0M 7s 2023-12-03T11:39:24.267 INFO:teuthology.orchestra.run.smithi012.stderr: 37850K .......... .......... .......... .......... .......... 40% 14.3M 7s 2023-12-03T11:39:24.271 INFO:teuthology.orchestra.run.smithi012.stderr: 37900K .......... .......... .......... .......... .......... 40% 15.2M 7s 2023-12-03T11:39:24.275 INFO:teuthology.orchestra.run.smithi012.stderr: 37950K .......... .......... .......... .......... .......... 40% 21.0M 7s 2023-12-03T11:39:24.279 INFO:teuthology.orchestra.run.smithi012.stderr: 38000K .......... .......... .......... .......... .......... 40% 9.00M 7s 2023-12-03T11:39:24.281 INFO:teuthology.orchestra.run.smithi012.stderr: 38050K .......... .......... .......... .......... .......... 41% 19.9M 7s 2023-12-03T11:39:24.285 INFO:teuthology.orchestra.run.smithi012.stderr: 38100K .......... .......... .......... .......... .......... 41% 10.6M 7s 2023-12-03T11:39:24.287 INFO:teuthology.orchestra.run.smithi012.stderr: 38150K .......... .......... .......... .......... .......... 41% 31.1M 7s 2023-12-03T11:39:24.291 INFO:teuthology.orchestra.run.smithi012.stderr: 38200K .......... .......... .......... .......... .......... 41% 11.5M 7s 2023-12-03T11:39:24.294 INFO:teuthology.orchestra.run.smithi012.stderr: 38250K .......... .......... .......... .......... .......... 41% 19.6M 7s 2023-12-03T11:39:24.297 INFO:teuthology.orchestra.run.smithi012.stderr: 38300K .......... .......... .......... .......... .......... 41% 13.7M 7s 2023-12-03T11:39:24.302 INFO:teuthology.orchestra.run.smithi012.stderr: 38350K .......... .......... .......... .......... .......... 41% 13.6M 7s 2023-12-03T11:39:24.305 INFO:teuthology.orchestra.run.smithi012.stderr: 38400K .......... .......... .......... .......... .......... 41% 11.6M 7s 2023-12-03T11:39:24.308 INFO:teuthology.orchestra.run.smithi012.stderr: 38450K .......... .......... .......... .......... .......... 41% 15.4M 7s 2023-12-03T11:39:24.310 INFO:teuthology.orchestra.run.smithi012.stderr: 38500K .......... .......... .......... .......... .......... 41% 24.1M 7s 2023-12-03T11:39:24.315 INFO:teuthology.orchestra.run.smithi012.stderr: 38550K .......... .......... .......... .......... .......... 41% 11.3M 7s 2023-12-03T11:39:24.317 INFO:teuthology.orchestra.run.smithi012.stderr: 38600K .......... .......... .......... .......... .......... 41% 20.2M 7s 2023-12-03T11:39:24.321 INFO:teuthology.orchestra.run.smithi012.stderr: 38650K .......... .......... .......... .......... .......... 41% 11.9M 7s 2023-12-03T11:39:24.323 INFO:teuthology.orchestra.run.smithi012.stderr: 38700K .......... .......... .......... .......... .......... 41% 22.7M 7s 2023-12-03T11:39:24.327 INFO:teuthology.orchestra.run.smithi012.stderr: 38750K .......... .......... .......... .......... .......... 41% 19.9M 7s 2023-12-03T11:39:24.331 INFO:teuthology.orchestra.run.smithi012.stderr: 38800K .......... .......... .......... .......... .......... 41% 10.1M 7s 2023-12-03T11:39:24.335 INFO:teuthology.orchestra.run.smithi012.stderr: 38850K .......... .......... .......... .......... .......... 41% 12.0M 7s 2023-12-03T11:39:24.337 INFO:teuthology.orchestra.run.smithi012.stderr: 38900K .......... .......... .......... .......... .......... 41% 19.5M 7s 2023-12-03T11:39:24.340 INFO:teuthology.orchestra.run.smithi012.stderr: 38950K .......... .......... .......... .......... .......... 41% 19.9M 7s 2023-12-03T11:39:24.342 INFO:teuthology.orchestra.run.smithi012.stderr: 39000K .......... .......... .......... .......... .......... 42% 16.7M 7s 2023-12-03T11:39:24.346 INFO:teuthology.orchestra.run.smithi012.stderr: 39050K .......... .......... .......... .......... .......... 42% 15.0M 7s 2023-12-03T11:39:24.350 INFO:teuthology.orchestra.run.smithi012.stderr: 39100K .......... .......... .......... .......... .......... 42% 12.2M 7s 2023-12-03T11:39:24.354 INFO:teuthology.orchestra.run.smithi012.stderr: 39150K .......... .......... .......... .......... .......... 42% 19.0M 7s 2023-12-03T11:39:24.356 INFO:teuthology.orchestra.run.smithi012.stderr: 39200K .......... .......... .......... .......... .......... 42% 14.8M 7s 2023-12-03T11:39:24.360 INFO:teuthology.orchestra.run.smithi012.stderr: 39250K .......... .......... .......... .......... .......... 42% 10.6M 7s 2023-12-03T11:39:24.362 INFO:teuthology.orchestra.run.smithi012.stderr: 39300K .......... .......... .......... .......... .......... 42% 22.2M 7s 2023-12-03T11:39:24.366 INFO:teuthology.orchestra.run.smithi012.stderr: 39350K .......... .......... .......... .......... .......... 42% 13.2M 7s 2023-12-03T11:39:24.369 INFO:teuthology.orchestra.run.smithi012.stderr: 39400K .......... .......... .......... .......... .......... 42% 16.6M 7s 2023-12-03T11:39:24.371 INFO:teuthology.orchestra.run.smithi012.stderr: 39450K .......... .......... .......... .......... .......... 42% 25.1M 7s 2023-12-03T11:39:24.375 INFO:teuthology.orchestra.run.smithi012.stderr: 39500K .......... .......... .......... .......... .......... 42% 10.8M 7s 2023-12-03T11:39:24.379 INFO:teuthology.orchestra.run.smithi012.stderr: 39550K .......... .......... .......... .......... .......... 42% 13.2M 7s 2023-12-03T11:39:24.382 INFO:teuthology.orchestra.run.smithi012.stderr: 39600K .......... .......... .......... .......... .......... 42% 15.2M 7s 2023-12-03T11:39:24.385 INFO:teuthology.orchestra.run.smithi012.stderr: 39650K .......... .......... .......... .......... .......... 42% 22.3M 7s 2023-12-03T11:39:24.388 INFO:teuthology.orchestra.run.smithi012.stderr: 39700K .......... .......... .......... .......... .......... 42% 16.7M 7s 2023-12-03T11:39:24.391 INFO:teuthology.orchestra.run.smithi012.stderr: 39750K .......... .......... .......... .......... .......... 42% 13.4M 7s 2023-12-03T11:39:24.396 INFO:teuthology.orchestra.run.smithi012.stderr: 39800K .......... .......... .......... .......... .......... 42% 9.97M 7s 2023-12-03T11:39:24.398 INFO:teuthology.orchestra.run.smithi012.stderr: 39850K .......... .......... .......... .......... .......... 42% 36.0M 7s 2023-12-03T11:39:24.400 INFO:teuthology.orchestra.run.smithi012.stderr: 39900K .......... .......... .......... .......... .......... 43% 20.0M 7s 2023-12-03T11:39:24.405 INFO:teuthology.orchestra.run.smithi012.stderr: 39950K .......... .......... .......... .......... .......... 43% 16.8M 7s 2023-12-03T11:39:24.407 INFO:teuthology.orchestra.run.smithi012.stderr: 40000K .......... .......... .......... .......... .......... 43% 11.6M 7s 2023-12-03T11:39:24.412 INFO:teuthology.orchestra.run.smithi012.stderr: 40050K .......... .......... .......... .......... .......... 43% 10.9M 7s 2023-12-03T11:39:24.414 INFO:teuthology.orchestra.run.smithi012.stderr: 40100K .......... .......... .......... .......... .......... 43% 20.9M 7s 2023-12-03T11:39:24.416 INFO:teuthology.orchestra.run.smithi012.stderr: 40150K .......... .......... .......... .......... .......... 43% 25.0M 7s 2023-12-03T11:39:24.421 INFO:teuthology.orchestra.run.smithi012.stderr: 40200K .......... .......... .......... .......... .......... 43% 9.42M 7s 2023-12-03T11:39:24.424 INFO:teuthology.orchestra.run.smithi012.stderr: 40250K .......... .......... .......... .......... .......... 43% 18.9M 7s 2023-12-03T11:39:24.426 INFO:teuthology.orchestra.run.smithi012.stderr: 40300K .......... .......... .......... .......... .......... 43% 16.7M 7s 2023-12-03T11:39:24.430 INFO:teuthology.orchestra.run.smithi012.stderr: 40350K .......... .......... .......... .......... .......... 43% 20.7M 7s 2023-12-03T11:39:24.435 INFO:teuthology.orchestra.run.smithi012.stderr: 40400K .......... .......... .......... .......... .......... 43% 8.82M 7s 2023-12-03T11:39:24.437 INFO:teuthology.orchestra.run.smithi012.stderr: 40450K .......... .......... .......... .......... .......... 43% 19.3M 7s 2023-12-03T11:39:24.441 INFO:teuthology.orchestra.run.smithi012.stderr: 40500K .......... .......... .......... .......... .......... 43% 12.4M 7s 2023-12-03T11:39:24.443 INFO:teuthology.orchestra.run.smithi012.stderr: 40550K .......... .......... .......... .......... .......... 43% 26.3M 7s 2023-12-03T11:39:24.445 INFO:teuthology.orchestra.run.smithi012.stderr: 40600K .......... .......... .......... .......... .......... 43% 20.9M 7s 2023-12-03T11:39:24.449 INFO:teuthology.orchestra.run.smithi012.stderr: 40650K .......... .......... .......... .......... .......... 43% 11.8M 7s 2023-12-03T11:39:24.452 INFO:teuthology.orchestra.run.smithi012.stderr: 40700K .......... .......... .......... .......... .......... 43% 16.3M 7s 2023-12-03T11:39:24.457 INFO:teuthology.orchestra.run.smithi012.stderr: 40750K .......... .......... .......... .......... .......... 43% 11.7M 7s 2023-12-03T11:39:24.459 INFO:teuthology.orchestra.run.smithi012.stderr: 40800K .......... .......... .......... .......... .......... 43% 22.4M 7s 2023-12-03T11:39:24.462 INFO:teuthology.orchestra.run.smithi012.stderr: 40850K .......... .......... .......... .......... .......... 44% 14.4M 7s 2023-12-03T11:39:24.465 INFO:teuthology.orchestra.run.smithi012.stderr: 40900K .......... .......... .......... .......... .......... 44% 14.1M 7s 2023-12-03T11:39:24.468 INFO:teuthology.orchestra.run.smithi012.stderr: 40950K .......... .......... .......... .......... .......... 44% 16.8M 7s 2023-12-03T11:39:24.472 INFO:teuthology.orchestra.run.smithi012.stderr: 41000K .......... .......... .......... .......... .......... 44% 13.8M 7s 2023-12-03T11:39:24.475 INFO:teuthology.orchestra.run.smithi012.stderr: 41050K .......... .......... .......... .......... .......... 44% 16.4M 7s 2023-12-03T11:39:24.477 INFO:teuthology.orchestra.run.smithi012.stderr: 41100K .......... .......... .......... .......... .......... 44% 24.5M 7s 2023-12-03T11:39:24.482 INFO:teuthology.orchestra.run.smithi012.stderr: 41150K .......... .......... .......... .......... .......... 44% 13.7M 7s 2023-12-03T11:39:24.485 INFO:teuthology.orchestra.run.smithi012.stderr: 41200K .......... .......... .......... .......... .......... 44% 10.6M 7s 2023-12-03T11:39:24.488 INFO:teuthology.orchestra.run.smithi012.stderr: 41250K .......... .......... .......... .......... .......... 44% 16.8M 7s 2023-12-03T11:39:24.490 INFO:teuthology.orchestra.run.smithi012.stderr: 41300K .......... .......... .......... .......... .......... 44% 20.0M 7s 2023-12-03T11:39:24.494 INFO:teuthology.orchestra.run.smithi012.stderr: 41350K .......... .......... .......... .......... .......... 44% 15.1M 7s 2023-12-03T11:39:24.496 INFO:teuthology.orchestra.run.smithi012.stderr: 41400K .......... .......... .......... .......... .......... 44% 17.2M 7s 2023-12-03T11:39:24.500 INFO:teuthology.orchestra.run.smithi012.stderr: 41450K .......... .......... .......... .......... .......... 44% 14.0M 7s 2023-12-03T11:39:24.503 INFO:teuthology.orchestra.run.smithi012.stderr: 41500K .......... .......... .......... .......... .......... 44% 15.5M 7s 2023-12-03T11:39:24.507 INFO:teuthology.orchestra.run.smithi012.stderr: 41550K .......... .......... .......... .......... .......... 44% 20.4M 7s 2023-12-03T11:39:24.510 INFO:teuthology.orchestra.run.smithi012.stderr: 41600K .......... .......... .......... .......... .......... 44% 12.4M 7s 2023-12-03T11:39:24.513 INFO:teuthology.orchestra.run.smithi012.stderr: 41650K .......... .......... .......... .......... .......... 44% 13.6M 7s 2023-12-03T11:39:24.517 INFO:teuthology.orchestra.run.smithi012.stderr: 41700K .......... .......... .......... .......... .......... 44% 14.1M 7s 2023-12-03T11:39:24.519 INFO:teuthology.orchestra.run.smithi012.stderr: 41750K .......... .......... .......... .......... .......... 44% 17.2M 7s 2023-12-03T11:39:24.521 INFO:teuthology.orchestra.run.smithi012.stderr: 41800K .......... .......... .......... .......... .......... 45% 22.1M 7s 2023-12-03T11:39:24.524 INFO:teuthology.orchestra.run.smithi012.stderr: 41850K .......... .......... .......... .......... .......... 45% 16.1M 7s 2023-12-03T11:39:24.528 INFO:teuthology.orchestra.run.smithi012.stderr: 41900K .......... .......... .......... .......... .......... 45% 12.5M 7s 2023-12-03T11:39:24.533 INFO:teuthology.orchestra.run.smithi012.stderr: 41950K .......... .......... .......... .......... .......... 45% 17.3M 7s 2023-12-03T11:39:24.535 INFO:teuthology.orchestra.run.smithi012.stderr: 42000K .......... .......... .......... .......... .......... 45% 13.4M 7s 2023-12-03T11:39:24.538 INFO:teuthology.orchestra.run.smithi012.stderr: 42050K .......... .......... .......... .......... .......... 45% 15.7M 7s 2023-12-03T11:39:24.540 INFO:teuthology.orchestra.run.smithi012.stderr: 42100K .......... .......... .......... .......... .......... 45% 22.3M 6s 2023-12-03T11:39:24.544 INFO:teuthology.orchestra.run.smithi012.stderr: 42150K .......... .......... .......... .......... .......... 45% 13.3M 6s 2023-12-03T11:39:24.548 INFO:teuthology.orchestra.run.smithi012.stderr: 42200K .......... .......... .......... .......... .......... 45% 13.2M 6s 2023-12-03T11:39:24.550 INFO:teuthology.orchestra.run.smithi012.stderr: 42250K .......... .......... .......... .......... .......... 45% 22.3M 6s 2023-12-03T11:39:24.553 INFO:teuthology.orchestra.run.smithi012.stderr: 42300K .......... .......... .......... .......... .......... 45% 14.4M 6s 2023-12-03T11:39:24.558 INFO:teuthology.orchestra.run.smithi012.stderr: 42350K .......... .......... .......... .......... .......... 45% 21.1M 6s 2023-12-03T11:39:24.560 INFO:teuthology.orchestra.run.smithi012.stderr: 42400K .......... .......... .......... .......... .......... 45% 12.1M 6s 2023-12-03T11:39:24.563 INFO:teuthology.orchestra.run.smithi012.stderr: 42450K .......... .......... .......... .......... .......... 45% 14.4M 6s 2023-12-03T11:39:24.566 INFO:teuthology.orchestra.run.smithi012.stderr: 42500K .......... .......... .......... .......... .......... 45% 16.4M 6s 2023-12-03T11:39:24.569 INFO:teuthology.orchestra.run.smithi012.stderr: 42550K .......... .......... .......... .......... .......... 45% 16.8M 6s 2023-12-03T11:39:24.572 INFO:teuthology.orchestra.run.smithi012.stderr: 42600K .......... .......... .......... .......... .......... 45% 16.5M 6s 2023-12-03T11:39:24.575 INFO:teuthology.orchestra.run.smithi012.stderr: 42650K .......... .......... .......... .......... .......... 45% 17.0M 6s 2023-12-03T11:39:24.578 INFO:teuthology.orchestra.run.smithi012.stderr: 42700K .......... .......... .......... .......... .......... 46% 15.9M 6s 2023-12-03T11:39:24.582 INFO:teuthology.orchestra.run.smithi012.stderr: 42750K .......... .......... .......... .......... .......... 46% 16.2M 6s 2023-12-03T11:39:24.585 INFO:teuthology.orchestra.run.smithi012.stderr: 42800K .......... .......... .......... .......... .......... 46% 12.8M 6s 2023-12-03T11:39:24.588 INFO:teuthology.orchestra.run.smithi012.stderr: 42850K .......... .......... .......... .......... .......... 46% 15.1M 6s 2023-12-03T11:39:24.591 INFO:teuthology.orchestra.run.smithi012.stderr: 42900K .......... .......... .......... .......... .......... 46% 14.8M 6s 2023-12-03T11:39:24.594 INFO:teuthology.orchestra.run.smithi012.stderr: 42950K .......... .......... .......... .......... .......... 46% 16.3M 6s 2023-12-03T11:39:24.598 INFO:teuthology.orchestra.run.smithi012.stderr: 43000K .......... .......... .......... .......... .......... 46% 13.6M 6s 2023-12-03T11:39:24.599 INFO:teuthology.orchestra.run.smithi012.stderr: 43050K .......... .......... .......... .......... .......... 46% 27.9M 6s 2023-12-03T11:39:24.603 INFO:teuthology.orchestra.run.smithi012.stderr: 43100K .......... .......... .......... .......... .......... 46% 15.4M 6s 2023-12-03T11:39:24.607 INFO:teuthology.orchestra.run.smithi012.stderr: 43150K .......... .......... .......... .......... .......... 46% 14.7M 6s 2023-12-03T11:39:24.609 INFO:teuthology.orchestra.run.smithi012.stderr: 43200K .......... .......... .......... .......... .......... 46% 14.5M 6s 2023-12-03T11:39:24.614 INFO:teuthology.orchestra.run.smithi012.stderr: 43250K .......... .......... .......... .......... .......... 46% 12.0M 6s 2023-12-03T11:39:24.615 INFO:teuthology.orchestra.run.smithi012.stderr: 43300K .......... .......... .......... .......... .......... 46% 28.8M 6s 2023-12-03T11:39:24.619 INFO:teuthology.orchestra.run.smithi012.stderr: 43350K .......... .......... .......... .......... .......... 46% 12.6M 6s 2023-12-03T11:39:24.622 INFO:teuthology.orchestra.run.smithi012.stderr: 43400K .......... .......... .......... .......... .......... 46% 14.2M 6s 2023-12-03T11:39:24.625 INFO:teuthology.orchestra.run.smithi012.stderr: 43450K .......... .......... .......... .......... .......... 46% 15.8M 6s 2023-12-03T11:39:24.627 INFO:teuthology.orchestra.run.smithi012.stderr: 43500K .......... .......... .......... .......... .......... 46% 27.4M 6s 2023-12-03T11:39:24.631 INFO:teuthology.orchestra.run.smithi012.stderr: 43550K .......... .......... .......... .......... .......... 46% 18.7M 6s 2023-12-03T11:39:24.635 INFO:teuthology.orchestra.run.smithi012.stderr: 43600K .......... .......... .......... .......... .......... 46% 10.7M 6s 2023-12-03T11:39:24.639 INFO:teuthology.orchestra.run.smithi012.stderr: 43650K .......... .......... .......... .......... .......... 47% 11.4M 6s 2023-12-03T11:39:24.641 INFO:teuthology.orchestra.run.smithi012.stderr: 43700K .......... .......... .......... .......... .......... 47% 17.5M 6s 2023-12-03T11:39:24.643 INFO:teuthology.orchestra.run.smithi012.stderr: 43750K .......... .......... .......... .......... .......... 47% 30.4M 6s 2023-12-03T11:39:24.646 INFO:teuthology.orchestra.run.smithi012.stderr: 43800K .......... .......... .......... .......... .......... 47% 17.0M 6s 2023-12-03T11:39:24.651 INFO:teuthology.orchestra.run.smithi012.stderr: 43850K .......... .......... .......... .......... .......... 47% 10.0M 6s 2023-12-03T11:39:24.654 INFO:teuthology.orchestra.run.smithi012.stderr: 43900K .......... .......... .......... .......... .......... 47% 17.6M 6s 2023-12-03T11:39:24.657 INFO:teuthology.orchestra.run.smithi012.stderr: 43950K .......... .......... .......... .......... .......... 47% 25.9M 6s 2023-12-03T11:39:24.659 INFO:teuthology.orchestra.run.smithi012.stderr: 44000K .......... .......... .......... .......... .......... 47% 15.7M 6s 2023-12-03T11:39:24.665 INFO:teuthology.orchestra.run.smithi012.stderr: 44050K .......... .......... .......... .......... .......... 47% 8.40M 6s 2023-12-03T11:39:24.666 INFO:teuthology.orchestra.run.smithi012.stderr: 44100K .......... .......... .......... .......... .......... 47% 22.2M 6s 2023-12-03T11:39:24.669 INFO:teuthology.orchestra.run.smithi012.stderr: 44150K .......... .......... .......... .......... .......... 47% 20.8M 6s 2023-12-03T11:39:24.671 INFO:teuthology.orchestra.run.smithi012.stderr: 44200K .......... .......... .......... .......... .......... 47% 22.9M 6s 2023-12-03T11:39:24.674 INFO:teuthology.orchestra.run.smithi012.stderr: 44250K .......... .......... .......... .......... .......... 47% 15.4M 6s 2023-12-03T11:39:24.679 INFO:teuthology.orchestra.run.smithi012.stderr: 44300K .......... .......... .......... .......... .......... 47% 9.63M 6s 2023-12-03T11:39:24.684 INFO:teuthology.orchestra.run.smithi012.stderr: 44350K .......... .......... .......... .......... .......... 47% 14.6M 6s 2023-12-03T11:39:24.686 INFO:teuthology.orchestra.run.smithi012.stderr: 44400K .......... .......... .......... .......... .......... 47% 15.3M 6s 2023-12-03T11:39:24.689 INFO:teuthology.orchestra.run.smithi012.stderr: 44450K .......... .......... .......... .......... .......... 47% 17.3M 6s 2023-12-03T11:39:24.691 INFO:teuthology.orchestra.run.smithi012.stderr: 44500K .......... .......... .......... .......... .......... 47% 22.3M 6s 2023-12-03T11:39:24.695 INFO:teuthology.orchestra.run.smithi012.stderr: 44550K .......... .......... .......... .......... .......... 48% 12.6M 6s 2023-12-03T11:39:24.698 INFO:teuthology.orchestra.run.smithi012.stderr: 44600K .......... .......... .......... .......... .......... 48% 13.4M 6s 2023-12-03T11:39:24.701 INFO:teuthology.orchestra.run.smithi012.stderr: 44650K .......... .......... .......... .......... .......... 48% 20.6M 6s 2023-12-03T11:39:24.703 INFO:teuthology.orchestra.run.smithi012.stderr: 44700K .......... .......... .......... .......... .......... 48% 22.2M 6s 2023-12-03T11:39:24.706 INFO:teuthology.orchestra.run.smithi012.stderr: 44750K .......... .......... .......... .......... .......... 48% 20.5M 6s 2023-12-03T11:39:24.710 INFO:teuthology.orchestra.run.smithi012.stderr: 44800K .......... .......... .......... .......... .......... 48% 10.1M 6s 2023-12-03T11:39:24.713 INFO:teuthology.orchestra.run.smithi012.stderr: 44850K .......... .......... .......... .......... .......... 48% 15.3M 6s 2023-12-03T11:39:24.716 INFO:teuthology.orchestra.run.smithi012.stderr: 44900K .......... .......... .......... .......... .......... 48% 19.5M 6s 2023-12-03T11:39:24.718 INFO:teuthology.orchestra.run.smithi012.stderr: 44950K .......... .......... .......... .......... .......... 48% 21.8M 6s 2023-12-03T11:39:24.720 INFO:teuthology.orchestra.run.smithi012.stderr: 45000K .......... .......... .......... .......... .......... 48% 21.6M 6s 2023-12-03T11:39:24.725 INFO:teuthology.orchestra.run.smithi012.stderr: 45050K .......... .......... .......... .......... .......... 48% 11.5M 6s 2023-12-03T11:39:24.727 INFO:teuthology.orchestra.run.smithi012.stderr: 45100K .......... .......... .......... .......... .......... 48% 19.6M 6s 2023-12-03T11:39:24.731 INFO:teuthology.orchestra.run.smithi012.stderr: 45150K .......... .......... .......... .......... .......... 48% 15.4M 6s 2023-12-03T11:39:24.733 INFO:teuthology.orchestra.run.smithi012.stderr: 45200K .......... .......... .......... .......... .......... 48% 16.8M 6s 2023-12-03T11:39:24.735 INFO:teuthology.orchestra.run.smithi012.stderr: 45250K .......... .......... .......... .......... .......... 48% 23.4M 6s 2023-12-03T11:39:24.740 INFO:teuthology.orchestra.run.smithi012.stderr: 45300K .......... .......... .......... .......... .......... 48% 9.81M 6s 2023-12-03T11:39:24.742 INFO:teuthology.orchestra.run.smithi012.stderr: 45350K .......... .......... .......... .......... .......... 48% 27.5M 6s 2023-12-03T11:39:24.746 INFO:teuthology.orchestra.run.smithi012.stderr: 45400K .......... .......... .......... .......... .......... 48% 14.9M 6s 2023-12-03T11:39:24.747 INFO:teuthology.orchestra.run.smithi012.stderr: 45450K .......... .......... .......... .......... .......... 48% 27.4M 6s 2023-12-03T11:39:24.750 INFO:teuthology.orchestra.run.smithi012.stderr: 45500K .......... .......... .......... .......... .......... 49% 19.1M 6s 2023-12-03T11:39:24.757 INFO:teuthology.orchestra.run.smithi012.stderr: 45550K .......... .......... .......... .......... .......... 49% 10.4M 6s 2023-12-03T11:39:24.759 INFO:teuthology.orchestra.run.smithi012.stderr: 45600K .......... .......... .......... .......... .......... 49% 11.9M 6s 2023-12-03T11:39:24.760 INFO:teuthology.orchestra.run.smithi012.stderr: 45650K .......... .......... .......... .......... .......... 49% 25.3M 6s 2023-12-03T11:39:24.766 INFO:teuthology.orchestra.run.smithi012.stderr: 45700K .......... .......... .......... .......... .......... 49% 21.0M 6s 2023-12-03T11:39:24.766 INFO:teuthology.orchestra.run.smithi012.stderr: 45750K .......... .......... .......... .......... .......... 49% 18.2M 6s 2023-12-03T11:39:24.772 INFO:teuthology.orchestra.run.smithi012.stderr: 45800K .......... .......... .......... .......... .......... 49% 8.08M 6s 2023-12-03T11:39:24.773 INFO:teuthology.orchestra.run.smithi012.stderr: 45850K .......... .......... .......... .......... .......... 49% 35.8M 6s 2023-12-03T11:39:24.775 INFO:teuthology.orchestra.run.smithi012.stderr: 45900K .......... .......... .......... .......... .......... 49% 23.2M 6s 2023-12-03T11:39:24.778 INFO:teuthology.orchestra.run.smithi012.stderr: 45950K .......... .......... .......... .......... .......... 49% 21.6M 6s 2023-12-03T11:39:24.780 INFO:teuthology.orchestra.run.smithi012.stderr: 46000K .......... .......... .......... .......... .......... 49% 16.5M 6s 2023-12-03T11:39:24.786 INFO:teuthology.orchestra.run.smithi012.stderr: 46050K .......... .......... .......... .......... .......... 49% 7.80M 6s 2023-12-03T11:39:24.788 INFO:teuthology.orchestra.run.smithi012.stderr: 46100K .......... .......... .......... .......... .......... 49% 33.7M 6s 2023-12-03T11:39:24.790 INFO:teuthology.orchestra.run.smithi012.stderr: 46150K .......... .......... .......... .......... .......... 49% 21.6M 6s 2023-12-03T11:39:24.792 INFO:teuthology.orchestra.run.smithi012.stderr: 46200K .......... .......... .......... .......... .......... 49% 23.4M 6s 2023-12-03T11:39:24.796 INFO:teuthology.orchestra.run.smithi012.stderr: 46250K .......... .......... .......... .......... .......... 49% 13.4M 6s 2023-12-03T11:39:24.797 INFO:teuthology.orchestra.run.smithi012.stderr: 46300K .......... .......... .......... .......... .......... 49% 34.0M 6s 2023-12-03T11:39:24.803 INFO:teuthology.orchestra.run.smithi012.stderr: 46350K .......... .......... .......... .......... .......... 49% 8.96M 6s 2023-12-03T11:39:24.805 INFO:teuthology.orchestra.run.smithi012.stderr: 46400K .......... .......... .......... .......... .......... 49% 21.6M 6s 2023-12-03T11:39:24.808 INFO:teuthology.orchestra.run.smithi012.stderr: 46450K .......... .......... .......... .......... .......... 50% 16.6M 6s 2023-12-03T11:39:24.811 INFO:teuthology.orchestra.run.smithi012.stderr: 46500K .......... .......... .......... .......... .......... 50% 17.4M 6s 2023-12-03T11:39:24.812 INFO:teuthology.orchestra.run.smithi012.stderr: 46550K .......... .......... .......... .......... .......... 50% 30.0M 6s 2023-12-03T11:39:24.818 INFO:teuthology.orchestra.run.smithi012.stderr: 46600K .......... .......... .......... .......... .......... 50% 9.33M 6s 2023-12-03T11:39:24.820 INFO:teuthology.orchestra.run.smithi012.stderr: 46650K .......... .......... .......... .......... .......... 50% 20.6M 6s 2023-12-03T11:39:24.822 INFO:teuthology.orchestra.run.smithi012.stderr: 46700K .......... .......... .......... .......... .......... 50% 23.2M 6s 2023-12-03T11:39:24.825 INFO:teuthology.orchestra.run.smithi012.stderr: 46750K .......... .......... .......... .......... .......... 50% 23.3M 6s 2023-12-03T11:39:24.828 INFO:teuthology.orchestra.run.smithi012.stderr: 46800K .......... .......... .......... .......... .......... 50% 14.3M 6s 2023-12-03T11:39:24.833 INFO:teuthology.orchestra.run.smithi012.stderr: 46850K .......... .......... .......... .......... .......... 50% 9.54M 6s 2023-12-03T11:39:24.835 INFO:teuthology.orchestra.run.smithi012.stderr: 46900K .......... .......... .......... .......... .......... 50% 21.5M 6s 2023-12-03T11:39:24.837 INFO:teuthology.orchestra.run.smithi012.stderr: 46950K .......... .......... .......... .......... .......... 50% 22.6M 6s 2023-12-03T11:39:24.839 INFO:teuthology.orchestra.run.smithi012.stderr: 47000K .......... .......... .......... .......... .......... 50% 24.1M 6s 2023-12-03T11:39:24.843 INFO:teuthology.orchestra.run.smithi012.stderr: 47050K .......... .......... .......... .......... .......... 50% 14.9M 6s 2023-12-03T11:39:24.848 INFO:teuthology.orchestra.run.smithi012.stderr: 47100K .......... .......... .......... .......... .......... 50% 10.2M 6s 2023-12-03T11:39:24.850 INFO:teuthology.orchestra.run.smithi012.stderr: 47150K .......... .......... .......... .......... .......... 50% 23.7M 6s 2023-12-03T11:39:24.852 INFO:teuthology.orchestra.run.smithi012.stderr: 47200K .......... .......... .......... .......... .......... 50% 16.6M 6s 2023-12-03T11:39:24.855 INFO:teuthology.orchestra.run.smithi012.stderr: 47250K .......... .......... .......... .......... .......... 50% 22.8M 6s 2023-12-03T11:39:24.857 INFO:teuthology.orchestra.run.smithi012.stderr: 47300K .......... .......... .......... .......... .......... 50% 16.5M 5s 2023-12-03T11:39:24.862 INFO:teuthology.orchestra.run.smithi012.stderr: 47350K .......... .......... .......... .......... .......... 51% 10.5M 5s 2023-12-03T11:39:24.864 INFO:teuthology.orchestra.run.smithi012.stderr: 47400K .......... .......... .......... .......... .......... 51% 23.7M 5s 2023-12-03T11:39:24.866 INFO:teuthology.orchestra.run.smithi012.stderr: 47450K .......... .......... .......... .......... .......... 51% 23.4M 5s 2023-12-03T11:39:24.869 INFO:teuthology.orchestra.run.smithi012.stderr: 47500K .......... .......... .......... .......... .......... 51% 16.6M 5s 2023-12-03T11:39:24.872 INFO:teuthology.orchestra.run.smithi012.stderr: 47550K .......... .......... .......... .......... .......... 51% 18.6M 5s 2023-12-03T11:39:24.877 INFO:teuthology.orchestra.run.smithi012.stderr: 47600K .......... .......... .......... .......... .......... 51% 9.06M 5s 2023-12-03T11:39:24.880 INFO:teuthology.orchestra.run.smithi012.stderr: 47650K .......... .......... .......... .......... .......... 51% 18.3M 5s 2023-12-03T11:39:24.883 INFO:teuthology.orchestra.run.smithi012.stderr: 47700K .......... .......... .......... .......... .......... 51% 17.2M 5s 2023-12-03T11:39:24.885 INFO:teuthology.orchestra.run.smithi012.stderr: 47750K .......... .......... .......... .......... .......... 51% 23.8M 5s 2023-12-03T11:39:24.887 INFO:teuthology.orchestra.run.smithi012.stderr: 47800K .......... .......... .......... .......... .......... 51% 23.0M 5s 2023-12-03T11:39:24.890 INFO:teuthology.orchestra.run.smithi012.stderr: 47850K .......... .......... .......... .......... .......... 51% 13.2M 5s 2023-12-03T11:39:24.893 INFO:teuthology.orchestra.run.smithi012.stderr: 47900K .......... .......... .......... .......... .......... 51% 18.1M 5s 2023-12-03T11:39:24.897 INFO:teuthology.orchestra.run.smithi012.stderr: 47950K .......... .......... .......... .......... .......... 51% 13.4M 5s 2023-12-03T11:39:24.899 INFO:teuthology.orchestra.run.smithi012.stderr: 48000K .......... .......... .......... .......... .......... 51% 18.4M 5s 2023-12-03T11:39:24.902 INFO:teuthology.orchestra.run.smithi012.stderr: 48050K .......... .......... .......... .......... .......... 51% 22.9M 5s 2023-12-03T11:39:24.907 INFO:teuthology.orchestra.run.smithi012.stderr: 48100K .......... .......... .......... .......... .......... 51% 9.36M 5s 2023-12-03T11:39:24.908 INFO:teuthology.orchestra.run.smithi012.stderr: 48150K .......... .......... .......... .......... .......... 51% 34.5M 5s 2023-12-03T11:39:24.912 INFO:teuthology.orchestra.run.smithi012.stderr: 48200K .......... .......... .......... .......... .......... 51% 13.5M 5s 2023-12-03T11:39:24.913 INFO:teuthology.orchestra.run.smithi012.stderr: 48250K .......... .......... .......... .......... .......... 51% 29.5M 5s 2023-12-03T11:39:24.916 INFO:teuthology.orchestra.run.smithi012.stderr: 48300K .......... .......... .......... .......... .......... 52% 22.9M 5s 2023-12-03T11:39:24.920 INFO:teuthology.orchestra.run.smithi012.stderr: 48350K .......... .......... .......... .......... .......... 52% 11.4M 5s 2023-12-03T11:39:24.924 INFO:teuthology.orchestra.run.smithi012.stderr: 48400K .......... .......... .......... .......... .......... 52% 11.2M 5s 2023-12-03T11:39:24.927 INFO:teuthology.orchestra.run.smithi012.stderr: 48450K .......... .......... .......... .......... .......... 52% 17.4M 5s 2023-12-03T11:39:24.929 INFO:teuthology.orchestra.run.smithi012.stderr: 48500K .......... .......... .......... .......... .......... 52% 28.3M 5s 2023-12-03T11:39:24.931 INFO:teuthology.orchestra.run.smithi012.stderr: 48550K .......... .......... .......... .......... .......... 52% 24.6M 5s 2023-12-03T11:39:24.935 INFO:teuthology.orchestra.run.smithi012.stderr: 48600K .......... .......... .......... .......... .......... 52% 11.9M 5s 2023-12-03T11:39:24.938 INFO:teuthology.orchestra.run.smithi012.stderr: 48650K .......... .......... .......... .......... .......... 52% 16.9M 5s 2023-12-03T11:39:24.940 INFO:teuthology.orchestra.run.smithi012.stderr: 48700K .......... .......... .......... .......... .......... 52% 17.4M 5s 2023-12-03T11:39:24.944 INFO:teuthology.orchestra.run.smithi012.stderr: 48750K .......... .......... .......... .......... .......... 52% 22.7M 5s 2023-12-03T11:39:24.946 INFO:teuthology.orchestra.run.smithi012.stderr: 48800K .......... .......... .......... .......... .......... 52% 16.0M 5s 2023-12-03T11:39:24.950 INFO:teuthology.orchestra.run.smithi012.stderr: 48850K .......... .......... .......... .......... .......... 52% 12.3M 5s 2023-12-03T11:39:24.953 INFO:teuthology.orchestra.run.smithi012.stderr: 48900K .......... .......... .......... .......... .......... 52% 17.1M 5s 2023-12-03T11:39:24.956 INFO:teuthology.orchestra.run.smithi012.stderr: 48950K .......... .......... .......... .......... .......... 52% 16.8M 5s 2023-12-03T11:39:24.959 INFO:teuthology.orchestra.run.smithi012.stderr: 49000K .......... .......... .......... .......... .......... 52% 15.7M 5s 2023-12-03T11:39:24.961 INFO:teuthology.orchestra.run.smithi012.stderr: 49050K .......... .......... .......... .......... .......... 52% 22.6M 5s 2023-12-03T11:39:24.963 INFO:teuthology.orchestra.run.smithi012.stderr: 49100K .......... .......... .......... .......... .......... 52% 19.8M 5s 2023-12-03T11:39:24.970 INFO:teuthology.orchestra.run.smithi012.stderr: 49150K .......... .......... .......... .......... .......... 52% 12.8M 5s 2023-12-03T11:39:24.970 INFO:teuthology.orchestra.run.smithi012.stderr: 49200K .......... .......... .......... .......... .......... 53% 14.6M 5s 2023-12-03T11:39:24.976 INFO:teuthology.orchestra.run.smithi012.stderr: 49250K .......... .......... .......... .......... .......... 53% 8.82M 5s 2023-12-03T11:39:24.978 INFO:teuthology.orchestra.run.smithi012.stderr: 49300K .......... .......... .......... .......... .......... 53% 29.0M 5s 2023-12-03T11:39:24.980 INFO:teuthology.orchestra.run.smithi012.stderr: 49350K .......... .......... .......... .......... .......... 53% 25.3M 5s 2023-12-03T11:39:24.982 INFO:teuthology.orchestra.run.smithi012.stderr: 49400K .......... .......... .......... .......... .......... 53% 22.2M 5s 2023-12-03T11:39:24.985 INFO:teuthology.orchestra.run.smithi012.stderr: 49450K .......... .......... .......... .......... .......... 53% 14.7M 5s 2023-12-03T11:39:24.987 INFO:teuthology.orchestra.run.smithi012.stderr: 49500K .......... .......... .......... .......... .......... 53% 22.0M 5s 2023-12-03T11:39:24.992 INFO:teuthology.orchestra.run.smithi012.stderr: 49550K .......... .......... .......... .......... .......... 53% 12.4M 5s 2023-12-03T11:39:24.994 INFO:teuthology.orchestra.run.smithi012.stderr: 49600K .......... .......... .......... .......... .......... 53% 15.7M 5s 2023-12-03T11:39:24.996 INFO:teuthology.orchestra.run.smithi012.stderr: 49650K .......... .......... .......... .......... .......... 53% 23.5M 5s 2023-12-03T11:39:25.000 INFO:teuthology.orchestra.run.smithi012.stderr: 49700K .......... .......... .......... .......... .......... 53% 13.8M 5s 2023-12-03T11:39:25.002 INFO:teuthology.orchestra.run.smithi012.stderr: 49750K .......... .......... .......... .......... .......... 53% 20.6M 5s 2023-12-03T11:39:25.006 INFO:teuthology.orchestra.run.smithi012.stderr: 49800K .......... .......... .......... .......... .......... 53% 12.3M 5s 2023-12-03T11:39:25.009 INFO:teuthology.orchestra.run.smithi012.stderr: 49850K .......... .......... .......... .......... .......... 53% 17.6M 5s 2023-12-03T11:39:25.011 INFO:teuthology.orchestra.run.smithi012.stderr: 49900K .......... .......... .......... .......... .......... 53% 21.6M 5s 2023-12-03T11:39:25.015 INFO:teuthology.orchestra.run.smithi012.stderr: 49950K .......... .......... .......... .......... .......... 53% 20.3M 5s 2023-12-03T11:39:25.017 INFO:teuthology.orchestra.run.smithi012.stderr: 50000K .......... .......... .......... .......... .......... 53% 14.3M 5s 2023-12-03T11:39:25.021 INFO:teuthology.orchestra.run.smithi012.stderr: 50050K .......... .......... .......... .......... .......... 53% 12.5M 5s 2023-12-03T11:39:25.024 INFO:teuthology.orchestra.run.smithi012.stderr: 50100K .......... .......... .......... .......... .......... 53% 17.5M 5s 2023-12-03T11:39:25.026 INFO:teuthology.orchestra.run.smithi012.stderr: 50150K .......... .......... .......... .......... .......... 54% 20.0M 5s 2023-12-03T11:39:25.029 INFO:teuthology.orchestra.run.smithi012.stderr: 50200K .......... .......... .......... .......... .......... 54% 19.3M 5s 2023-12-03T11:39:25.031 INFO:teuthology.orchestra.run.smithi012.stderr: 50250K .......... .......... .......... .......... .......... 54% 19.1M 5s 2023-12-03T11:39:25.036 INFO:teuthology.orchestra.run.smithi012.stderr: 50300K .......... .......... .......... .......... .......... 54% 10.8M 5s 2023-12-03T11:39:25.039 INFO:teuthology.orchestra.run.smithi012.stderr: 50350K .......... .......... .......... .......... .......... 54% 20.9M 5s 2023-12-03T11:39:25.041 INFO:teuthology.orchestra.run.smithi012.stderr: 50400K .......... .......... .......... .......... .......... 54% 17.8M 5s 2023-12-03T11:39:25.044 INFO:teuthology.orchestra.run.smithi012.stderr: 50450K .......... .......... .......... .......... .......... 54% 18.7M 5s 2023-12-03T11:39:25.049 INFO:teuthology.orchestra.run.smithi012.stderr: 50500K .......... .......... .......... .......... .......... 54% 9.66M 5s 2023-12-03T11:39:25.051 INFO:teuthology.orchestra.run.smithi012.stderr: 50550K .......... .......... .......... .......... .......... 54% 23.9M 5s 2023-12-03T11:39:25.053 INFO:teuthology.orchestra.run.smithi012.stderr: 50600K .......... .......... .......... .......... .......... 54% 21.2M 5s 2023-12-03T11:39:25.055 INFO:teuthology.orchestra.run.smithi012.stderr: 50650K .......... .......... .......... .......... .......... 54% 23.7M 5s 2023-12-03T11:39:25.058 INFO:teuthology.orchestra.run.smithi012.stderr: 50700K .......... .......... .......... .......... .......... 54% 17.0M 5s 2023-12-03T11:39:25.064 INFO:teuthology.orchestra.run.smithi012.stderr: 50750K .......... .......... .......... .......... .......... 54% 11.1M 5s 2023-12-03T11:39:25.065 INFO:teuthology.orchestra.run.smithi012.stderr: 50800K .......... .......... .......... .......... .......... 54% 15.7M 5s 2023-12-03T11:39:25.069 INFO:teuthology.orchestra.run.smithi012.stderr: 50850K .......... .......... .......... .......... .......... 54% 15.2M 5s 2023-12-03T11:39:25.071 INFO:teuthology.orchestra.run.smithi012.stderr: 50900K .......... .......... .......... .......... .......... 54% 23.7M 5s 2023-12-03T11:39:25.074 INFO:teuthology.orchestra.run.smithi012.stderr: 50950K .......... .......... .......... .......... .......... 54% 17.6M 5s 2023-12-03T11:39:25.077 INFO:teuthology.orchestra.run.smithi012.stderr: 51000K .......... .......... .......... .......... .......... 54% 13.0M 5s 2023-12-03T11:39:25.079 INFO:teuthology.orchestra.run.smithi012.stderr: 51050K .......... .......... .......... .......... .......... 55% 21.5M 5s 2023-12-03T11:39:25.083 INFO:teuthology.orchestra.run.smithi012.stderr: 51100K .......... .......... .......... .......... .......... 55% 13.9M 5s 2023-12-03T11:39:25.085 INFO:teuthology.orchestra.run.smithi012.stderr: 51150K .......... .......... .......... .......... .......... 55% 20.7M 5s 2023-12-03T11:39:25.089 INFO:teuthology.orchestra.run.smithi012.stderr: 51200K .......... .......... .......... .......... .......... 55% 16.3M 5s 2023-12-03T11:39:25.092 INFO:teuthology.orchestra.run.smithi012.stderr: 51250K .......... .......... .......... .......... .......... 55% 13.1M 5s 2023-12-03T11:39:25.094 INFO:teuthology.orchestra.run.smithi012.stderr: 51300K .......... .......... .......... .......... .......... 55% 21.7M 5s 2023-12-03T11:39:25.099 INFO:teuthology.orchestra.run.smithi012.stderr: 51350K .......... .......... .......... .......... .......... 55% 13.4M 5s 2023-12-03T11:39:25.101 INFO:teuthology.orchestra.run.smithi012.stderr: 51400K .......... .......... .......... .......... .......... 55% 22.2M 5s 2023-12-03T11:39:25.102 INFO:teuthology.orchestra.run.smithi012.stderr: 51450K .......... .......... .......... .......... .......... 55% 25.9M 5s 2023-12-03T11:39:25.108 INFO:teuthology.orchestra.run.smithi012.stderr: 51500K .......... .......... .......... .......... .......... 55% 10.6M 5s 2023-12-03T11:39:25.109 INFO:teuthology.orchestra.run.smithi012.stderr: 51550K .......... .......... .......... .......... .......... 55% 31.7M 5s 2023-12-03T11:39:25.113 INFO:teuthology.orchestra.run.smithi012.stderr: 51600K .......... .......... .......... .......... .......... 55% 11.3M 5s 2023-12-03T11:39:25.115 INFO:teuthology.orchestra.run.smithi012.stderr: 51650K .......... .......... .......... .......... .......... 55% 20.5M 5s 2023-12-03T11:39:25.117 INFO:teuthology.orchestra.run.smithi012.stderr: 51700K .......... .......... .......... .......... .......... 55% 26.1M 5s 2023-12-03T11:39:25.122 INFO:teuthology.orchestra.run.smithi012.stderr: 51750K .......... .......... .......... .......... .......... 55% 10.3M 5s 2023-12-03T11:39:25.125 INFO:teuthology.orchestra.run.smithi012.stderr: 51800K .......... .......... .......... .......... .......... 55% 18.6M 5s 2023-12-03T11:39:25.126 INFO:teuthology.orchestra.run.smithi012.stderr: 51850K .......... .......... .......... .......... .......... 55% 28.4M 5s 2023-12-03T11:39:25.130 INFO:teuthology.orchestra.run.smithi012.stderr: 51900K .......... .......... .......... .......... .......... 55% 12.6M 5s 2023-12-03T11:39:25.132 INFO:teuthology.orchestra.run.smithi012.stderr: 51950K .......... .......... .......... .......... .......... 55% 35.4M 5s 2023-12-03T11:39:25.134 INFO:teuthology.orchestra.run.smithi012.stderr: 52000K .......... .......... .......... .......... .......... 56% 16.9M 5s 2023-12-03T11:39:25.139 INFO:teuthology.orchestra.run.smithi012.stderr: 52050K .......... .......... .......... .......... .......... 56% 10.3M 5s 2023-12-03T11:39:25.141 INFO:teuthology.orchestra.run.smithi012.stderr: 52100K .......... .......... .......... .......... .......... 56% 24.2M 5s 2023-12-03T11:39:25.144 INFO:teuthology.orchestra.run.smithi012.stderr: 52150K .......... .......... .......... .......... .......... 56% 15.0M 5s 2023-12-03T11:39:25.146 INFO:teuthology.orchestra.run.smithi012.stderr: 52200K .......... .......... .......... .......... .......... 56% 26.7M 5s 2023-12-03T11:39:25.149 INFO:teuthology.orchestra.run.smithi012.stderr: 52250K .......... .......... .......... .......... .......... 56% 19.0M 5s 2023-12-03T11:39:25.152 INFO:teuthology.orchestra.run.smithi012.stderr: 52300K .......... .......... .......... .......... .......... 56% 13.6M 5s 2023-12-03T11:39:25.156 INFO:teuthology.orchestra.run.smithi012.stderr: 52350K .......... .......... .......... .......... .......... 56% 24.5M 5s 2023-12-03T11:39:25.158 INFO:teuthology.orchestra.run.smithi012.stderr: 52400K .......... .......... .......... .......... .......... 56% 14.7M 5s 2023-12-03T11:39:25.162 INFO:teuthology.orchestra.run.smithi012.stderr: 52450K .......... .......... .......... .......... .......... 56% 11.2M 5s 2023-12-03T11:39:25.164 INFO:teuthology.orchestra.run.smithi012.stderr: 52500K .......... .......... .......... .......... .......... 56% 23.4M 5s 2023-12-03T11:39:25.167 INFO:teuthology.orchestra.run.smithi012.stderr: 52550K .......... .......... .......... .......... .......... 56% 17.6M 5s 2023-12-03T11:39:25.169 INFO:teuthology.orchestra.run.smithi012.stderr: 52600K .......... .......... .......... .......... .......... 56% 20.6M 5s 2023-12-03T11:39:25.171 INFO:teuthology.orchestra.run.smithi012.stderr: 52650K .......... .......... .......... .......... .......... 56% 21.1M 5s 2023-12-03T11:39:25.175 INFO:teuthology.orchestra.run.smithi012.stderr: 52700K .......... .......... .......... .......... .......... 56% 14.4M 5s 2023-12-03T11:39:25.179 INFO:teuthology.orchestra.run.smithi012.stderr: 52750K .......... .......... .......... .......... .......... 56% 16.5M 5s 2023-12-03T11:39:25.181 INFO:teuthology.orchestra.run.smithi012.stderr: 52800K .......... .......... .......... .......... .......... 56% 17.0M 5s 2023-12-03T11:39:25.185 INFO:teuthology.orchestra.run.smithi012.stderr: 52850K .......... .......... .......... .......... .......... 56% 11.5M 5s 2023-12-03T11:39:25.187 INFO:teuthology.orchestra.run.smithi012.stderr: 52900K .......... .......... .......... .......... .......... 56% 21.6M 5s 2023-12-03T11:39:25.189 INFO:teuthology.orchestra.run.smithi012.stderr: 52950K .......... .......... .......... .......... .......... 57% 24.9M 5s 2023-12-03T11:39:25.192 INFO:teuthology.orchestra.run.smithi012.stderr: 53000K .......... .......... .......... .......... .......... 57% 17.6M 5s 2023-12-03T11:39:25.195 INFO:teuthology.orchestra.run.smithi012.stderr: 53050K .......... .......... .......... .......... .......... 57% 22.6M 5s 2023-12-03T11:39:25.197 INFO:teuthology.orchestra.run.smithi012.stderr: 53100K .......... .......... .......... .......... .......... 57% 16.0M 5s 2023-12-03T11:39:25.205 INFO:teuthology.orchestra.run.smithi012.stderr: 53150K .......... .......... .......... .......... .......... 57% 15.2M 5s 2023-12-03T11:39:25.207 INFO:teuthology.orchestra.run.smithi012.stderr: 53200K .......... .......... .......... .......... .......... 57% 7.33M 5s 2023-12-03T11:39:25.212 INFO:teuthology.orchestra.run.smithi012.stderr: 53250K .......... .......... .......... .......... .......... 57% 12.3M 5s 2023-12-03T11:39:25.212 INFO:teuthology.orchestra.run.smithi012.stderr: 53300K .......... .......... .......... .......... .......... 57% 56.4M 4s 2023-12-03T11:39:25.214 INFO:teuthology.orchestra.run.smithi012.stderr: 53350K .......... .......... .......... .......... .......... 57% 23.9M 4s 2023-12-03T11:39:25.216 INFO:teuthology.orchestra.run.smithi012.stderr: 53400K .......... .......... .......... .......... .......... 57% 23.8M 4s 2023-12-03T11:39:25.220 INFO:teuthology.orchestra.run.smithi012.stderr: 53450K .......... .......... .......... .......... .......... 57% 10.5M 4s 2023-12-03T11:39:25.223 INFO:teuthology.orchestra.run.smithi012.stderr: 53500K .......... .......... .......... .......... .......... 57% 19.5M 4s 2023-12-03T11:39:25.229 INFO:teuthology.orchestra.run.smithi012.stderr: 53550K .......... .......... .......... .......... .......... 57% 8.71M 4s 2023-12-03T11:39:25.232 INFO:teuthology.orchestra.run.smithi012.stderr: 53600K .......... .......... .......... .......... .......... 57% 14.9M 4s 2023-12-03T11:39:25.234 INFO:teuthology.orchestra.run.smithi012.stderr: 53650K .......... .......... .......... .......... .......... 57% 23.8M 4s 2023-12-03T11:39:25.236 INFO:teuthology.orchestra.run.smithi012.stderr: 53700K .......... .......... .......... .......... .......... 57% 23.6M 4s 2023-12-03T11:39:25.239 INFO:teuthology.orchestra.run.smithi012.stderr: 53750K .......... .......... .......... .......... .......... 57% 17.8M 4s 2023-12-03T11:39:25.243 INFO:teuthology.orchestra.run.smithi012.stderr: 53800K .......... .......... .......... .......... .......... 57% 10.7M 4s 2023-12-03T11:39:25.245 INFO:teuthology.orchestra.run.smithi012.stderr: 53850K .......... .......... .......... .......... .......... 58% 26.4M 4s 2023-12-03T11:39:25.248 INFO:teuthology.orchestra.run.smithi012.stderr: 53900K .......... .......... .......... .......... .......... 58% 18.9M 4s 2023-12-03T11:39:25.250 INFO:teuthology.orchestra.run.smithi012.stderr: 53950K .......... .......... .......... .......... .......... 58% 21.6M 4s 2023-12-03T11:39:25.253 INFO:teuthology.orchestra.run.smithi012.stderr: 54000K .......... .......... .......... .......... .......... 58% 17.0M 4s 2023-12-03T11:39:25.256 INFO:teuthology.orchestra.run.smithi012.stderr: 54050K .......... .......... .......... .......... .......... 58% 14.3M 4s 2023-12-03T11:39:25.260 INFO:teuthology.orchestra.run.smithi012.stderr: 54100K .......... .......... .......... .......... .......... 58% 15.8M 4s 2023-12-03T11:39:25.262 INFO:teuthology.orchestra.run.smithi012.stderr: 54150K .......... .......... .......... .......... .......... 58% 20.3M 4s 2023-12-03T11:39:25.264 INFO:teuthology.orchestra.run.smithi012.stderr: 54200K .......... .......... .......... .......... .......... 58% 21.2M 4s 2023-12-03T11:39:25.266 INFO:teuthology.orchestra.run.smithi012.stderr: 54250K .......... .......... .......... .......... .......... 58% 24.3M 4s 2023-12-03T11:39:25.268 INFO:teuthology.orchestra.run.smithi012.stderr: 54300K .......... .......... .......... .......... .......... 58% 26.6M 4s 2023-12-03T11:39:25.274 INFO:teuthology.orchestra.run.smithi012.stderr: 54350K .......... .......... .......... .......... .......... 58% 9.79M 4s 2023-12-03T11:39:25.275 INFO:teuthology.orchestra.run.smithi012.stderr: 54400K .......... .......... .......... .......... .......... 58% 21.9M 4s 2023-12-03T11:39:25.278 INFO:teuthology.orchestra.run.smithi012.stderr: 54450K .......... .......... .......... .......... .......... 58% 16.4M 4s 2023-12-03T11:39:25.280 INFO:teuthology.orchestra.run.smithi012.stderr: 54500K .......... .......... .......... .......... .......... 58% 23.8M 4s 2023-12-03T11:39:25.282 INFO:teuthology.orchestra.run.smithi012.stderr: 54550K .......... .......... .......... .......... .......... 58% 27.8M 4s 2023-12-03T11:39:25.286 INFO:teuthology.orchestra.run.smithi012.stderr: 54600K .......... .......... .......... .......... .......... 58% 12.0M 4s 2023-12-03T11:39:25.290 INFO:teuthology.orchestra.run.smithi012.stderr: 54650K .......... .......... .......... .......... .......... 58% 13.6M 4s 2023-12-03T11:39:25.292 INFO:teuthology.orchestra.run.smithi012.stderr: 54700K .......... .......... .......... .......... .......... 58% 22.2M 4s 2023-12-03T11:39:25.294 INFO:teuthology.orchestra.run.smithi012.stderr: 54750K .......... .......... .......... .......... .......... 58% 32.9M 4s 2023-12-03T11:39:25.296 INFO:teuthology.orchestra.run.smithi012.stderr: 54800K .......... .......... .......... .......... .......... 59% 16.5M 4s 2023-12-03T11:39:25.299 INFO:teuthology.orchestra.run.smithi012.stderr: 54850K .......... .......... .......... .......... .......... 59% 19.6M 4s 2023-12-03T11:39:25.304 INFO:teuthology.orchestra.run.smithi012.stderr: 54900K .......... .......... .......... .......... .......... 59% 9.63M 4s 2023-12-03T11:39:25.305 INFO:teuthology.orchestra.run.smithi012.stderr: 54950K .......... .......... .......... .......... .......... 59% 31.9M 4s 2023-12-03T11:39:25.307 INFO:teuthology.orchestra.run.smithi012.stderr: 55000K .......... .......... .......... .......... .......... 59% 25.6M 4s 2023-12-03T11:39:25.310 INFO:teuthology.orchestra.run.smithi012.stderr: 55050K .......... .......... .......... .......... .......... 59% 17.2M 4s 2023-12-03T11:39:25.313 INFO:teuthology.orchestra.run.smithi012.stderr: 55100K .......... .......... .......... .......... .......... 59% 17.9M 4s 2023-12-03T11:39:25.317 INFO:teuthology.orchestra.run.smithi012.stderr: 55150K .......... .......... .......... .......... .......... 59% 24.4M 4s 2023-12-03T11:39:25.319 INFO:teuthology.orchestra.run.smithi012.stderr: 55200K .......... .......... .......... .......... .......... 59% 10.3M 4s 2023-12-03T11:39:25.322 INFO:teuthology.orchestra.run.smithi012.stderr: 55250K .......... .......... .......... .......... .......... 59% 22.6M 4s 2023-12-03T11:39:25.324 INFO:teuthology.orchestra.run.smithi012.stderr: 55300K .......... .......... .......... .......... .......... 59% 19.2M 4s 2023-12-03T11:39:25.327 INFO:teuthology.orchestra.run.smithi012.stderr: 55350K .......... .......... .......... .......... .......... 59% 18.1M 4s 2023-12-03T11:39:25.329 INFO:teuthology.orchestra.run.smithi012.stderr: 55400K .......... .......... .......... .......... .......... 59% 21.9M 4s 2023-12-03T11:39:25.334 INFO:teuthology.orchestra.run.smithi012.stderr: 55450K .......... .......... .......... .......... .......... 59% 11.2M 4s 2023-12-03T11:39:25.335 INFO:teuthology.orchestra.run.smithi012.stderr: 55500K .......... .......... .......... .......... .......... 59% 27.8M 4s 2023-12-03T11:39:25.338 INFO:teuthology.orchestra.run.smithi012.stderr: 55550K .......... .......... .......... .......... .......... 59% 25.2M 4s 2023-12-03T11:39:25.340 INFO:teuthology.orchestra.run.smithi012.stderr: 55600K .......... .......... .......... .......... .......... 59% 17.7M 4s 2023-12-03T11:39:25.343 INFO:teuthology.orchestra.run.smithi012.stderr: 55650K .......... .......... .......... .......... .......... 59% 14.2M 4s 2023-12-03T11:39:25.347 INFO:teuthology.orchestra.run.smithi012.stderr: 55700K .......... .......... .......... .......... .......... 60% 13.7M 4s 2023-12-03T11:39:25.352 INFO:teuthology.orchestra.run.smithi012.stderr: 55750K .......... .......... .......... .......... .......... 60% 10.9M 4s 2023-12-03T11:39:25.353 INFO:teuthology.orchestra.run.smithi012.stderr: 55800K .......... .......... .......... .......... .......... 60% 31.8M 4s 2023-12-03T11:39:25.355 INFO:teuthology.orchestra.run.smithi012.stderr: 55850K .......... .......... .......... .......... .......... 60% 25.2M 4s 2023-12-03T11:39:25.358 INFO:teuthology.orchestra.run.smithi012.stderr: 55900K .......... .......... .......... .......... .......... 60% 18.3M 4s 2023-12-03T11:39:25.360 INFO:teuthology.orchestra.run.smithi012.stderr: 55950K .......... .......... .......... .......... .......... 60% 25.3M 4s 2023-12-03T11:39:25.363 INFO:teuthology.orchestra.run.smithi012.stderr: 56000K .......... .......... .......... .......... .......... 60% 13.2M 4s 2023-12-03T11:39:25.367 INFO:teuthology.orchestra.run.smithi012.stderr: 56050K .......... .......... .......... .......... .......... 60% 13.3M 4s 2023-12-03T11:39:25.369 INFO:teuthology.orchestra.run.smithi012.stderr: 56100K .......... .......... .......... .......... .......... 60% 23.5M 4s 2023-12-03T11:39:25.372 INFO:teuthology.orchestra.run.smithi012.stderr: 56150K .......... .......... .......... .......... .......... 60% 17.6M 4s 2023-12-03T11:39:25.373 INFO:teuthology.orchestra.run.smithi012.stderr: 56200K .......... .......... .......... .......... .......... 60% 26.4M 4s 2023-12-03T11:39:25.377 INFO:teuthology.orchestra.run.smithi012.stderr: 56250K .......... .......... .......... .......... .......... 60% 28.0M 4s 2023-12-03T11:39:25.381 INFO:teuthology.orchestra.run.smithi012.stderr: 56300K .......... .......... .......... .......... .......... 60% 8.59M 4s 2023-12-03T11:39:25.384 INFO:teuthology.orchestra.run.smithi012.stderr: 56350K .......... .......... .......... .......... .......... 60% 30.3M 4s 2023-12-03T11:39:25.386 INFO:teuthology.orchestra.run.smithi012.stderr: 56400K .......... .......... .......... .......... .......... 60% 17.0M 4s 2023-12-03T11:39:25.387 INFO:teuthology.orchestra.run.smithi012.stderr: 56450K .......... .......... .......... .......... .......... 60% 25.5M 4s 2023-12-03T11:39:25.389 INFO:teuthology.orchestra.run.smithi012.stderr: 56500K .......... .......... .......... .......... .......... 60% 26.6M 4s 2023-12-03T11:39:25.393 INFO:teuthology.orchestra.run.smithi012.stderr: 56550K .......... .......... .......... .......... .......... 60% 13.1M 4s 2023-12-03T11:39:25.396 INFO:teuthology.orchestra.run.smithi012.stderr: 56600K .......... .......... .......... .......... .......... 60% 13.9M 4s 2023-12-03T11:39:25.399 INFO:teuthology.orchestra.run.smithi012.stderr: 56650K .......... .......... .......... .......... .......... 61% 21.1M 4s 2023-12-03T11:39:25.400 INFO:teuthology.orchestra.run.smithi012.stderr: 56700K .......... .......... .......... .......... .......... 61% 29.5M 4s 2023-12-03T11:39:25.403 INFO:teuthology.orchestra.run.smithi012.stderr: 56750K .......... .......... .......... .......... .......... 61% 17.4M 4s 2023-12-03T11:39:25.407 INFO:teuthology.orchestra.run.smithi012.stderr: 56800K .......... .......... .......... .......... .......... 61% 16.0M 4s 2023-12-03T11:39:25.411 INFO:teuthology.orchestra.run.smithi012.stderr: 56850K .......... .......... .......... .......... .......... 61% 10.8M 4s 2023-12-03T11:39:25.412 INFO:teuthology.orchestra.run.smithi012.stderr: 56900K .......... .......... .......... .......... .......... 61% 32.6M 4s 2023-12-03T11:39:25.415 INFO:teuthology.orchestra.run.smithi012.stderr: 56950K .......... .......... .......... .......... .......... 61% 21.8M 4s 2023-12-03T11:39:25.417 INFO:teuthology.orchestra.run.smithi012.stderr: 57000K .......... .......... .......... .......... .......... 61% 19.4M 4s 2023-12-03T11:39:25.419 INFO:teuthology.orchestra.run.smithi012.stderr: 57050K .......... .......... .......... .......... .......... 61% 25.9M 4s 2023-12-03T11:39:25.421 INFO:teuthology.orchestra.run.smithi012.stderr: 57100K .......... .......... .......... .......... .......... 61% 21.2M 4s 2023-12-03T11:39:25.426 INFO:teuthology.orchestra.run.smithi012.stderr: 57150K .......... .......... .......... .......... .......... 61% 9.80M 4s 2023-12-03T11:39:25.429 INFO:teuthology.orchestra.run.smithi012.stderr: 57200K .......... .......... .......... .......... .......... 61% 21.6M 4s 2023-12-03T11:39:25.431 INFO:teuthology.orchestra.run.smithi012.stderr: 57250K .......... .......... .......... .......... .......... 61% 17.1M 4s 2023-12-03T11:39:25.433 INFO:teuthology.orchestra.run.smithi012.stderr: 57300K .......... .......... .......... .......... .......... 61% 31.0M 4s 2023-12-03T11:39:25.438 INFO:teuthology.orchestra.run.smithi012.stderr: 57350K .......... .......... .......... .......... .......... 61% 9.06M 4s 2023-12-03T11:39:25.442 INFO:teuthology.orchestra.run.smithi012.stderr: 57400K .......... .......... .......... .......... .......... 61% 16.7M 4s 2023-12-03T11:39:25.443 INFO:teuthology.orchestra.run.smithi012.stderr: 57450K .......... .......... .......... .......... .......... 61% 23.8M 4s 2023-12-03T11:39:25.445 INFO:teuthology.orchestra.run.smithi012.stderr: 57500K .......... .......... .......... .......... .......... 61% 27.7M 4s 2023-12-03T11:39:25.448 INFO:teuthology.orchestra.run.smithi012.stderr: 57550K .......... .......... .......... .......... .......... 62% 23.9M 4s 2023-12-03T11:39:25.454 INFO:teuthology.orchestra.run.smithi012.stderr: 57600K .......... .......... .......... .......... .......... 62% 8.03M 4s 2023-12-03T11:39:25.455 INFO:teuthology.orchestra.run.smithi012.stderr: 57650K .......... .......... .......... .......... .......... 62% 24.7M 4s 2023-12-03T11:39:25.457 INFO:teuthology.orchestra.run.smithi012.stderr: 57700K .......... .......... .......... .......... .......... 62% 30.0M 4s 2023-12-03T11:39:25.459 INFO:teuthology.orchestra.run.smithi012.stderr: 57750K .......... .......... .......... .......... .......... 62% 21.4M 4s 2023-12-03T11:39:25.461 INFO:teuthology.orchestra.run.smithi012.stderr: 57800K .......... .......... .......... .......... .......... 62% 27.9M 4s 2023-12-03T11:39:25.464 INFO:teuthology.orchestra.run.smithi012.stderr: 57850K .......... .......... .......... .......... .......... 62% 18.3M 4s 2023-12-03T11:39:25.468 INFO:teuthology.orchestra.run.smithi012.stderr: 57900K .......... .......... .......... .......... .......... 62% 10.5M 4s 2023-12-03T11:39:25.471 INFO:teuthology.orchestra.run.smithi012.stderr: 57950K .......... .......... .......... .......... .......... 62% 24.3M 4s 2023-12-03T11:39:25.473 INFO:teuthology.orchestra.run.smithi012.stderr: 58000K .......... .......... .......... .......... .......... 62% 16.5M 4s 2023-12-03T11:39:25.475 INFO:teuthology.orchestra.run.smithi012.stderr: 58050K .......... .......... .......... .......... .......... 62% 31.7M 4s 2023-12-03T11:39:25.477 INFO:teuthology.orchestra.run.smithi012.stderr: 58100K .......... .......... .......... .......... .......... 62% 18.8M 4s 2023-12-03T11:39:25.482 INFO:teuthology.orchestra.run.smithi012.stderr: 58150K .......... .......... .......... .......... .......... 62% 9.66M 4s 2023-12-03T11:39:25.484 INFO:teuthology.orchestra.run.smithi012.stderr: 58200K .......... .......... .......... .......... .......... 62% 30.5M 4s 2023-12-03T11:39:25.486 INFO:teuthology.orchestra.run.smithi012.stderr: 58250K .......... .......... .......... .......... .......... 62% 19.2M 4s 2023-12-03T11:39:25.489 INFO:teuthology.orchestra.run.smithi012.stderr: 58300K .......... .......... .......... .......... .......... 62% 20.3M 4s 2023-12-03T11:39:25.491 INFO:teuthology.orchestra.run.smithi012.stderr: 58350K .......... .......... .......... .......... .......... 62% 28.7M 4s 2023-12-03T11:39:25.494 INFO:teuthology.orchestra.run.smithi012.stderr: 58400K .......... .......... .......... .......... .......... 62% 16.0M 4s 2023-12-03T11:39:25.498 INFO:teuthology.orchestra.run.smithi012.stderr: 58450K .......... .......... .......... .......... .......... 62% 11.3M 4s 2023-12-03T11:39:25.500 INFO:teuthology.orchestra.run.smithi012.stderr: 58500K .......... .......... .......... .......... .......... 63% 21.0M 4s 2023-12-03T11:39:25.503 INFO:teuthology.orchestra.run.smithi012.stderr: 58550K .......... .......... .......... .......... .......... 63% 17.7M 4s 2023-12-03T11:39:25.504 INFO:teuthology.orchestra.run.smithi012.stderr: 58600K .......... .......... .......... .......... .......... 63% 34.7M 4s 2023-12-03T11:39:25.506 INFO:teuthology.orchestra.run.smithi012.stderr: 58650K .......... .......... .......... .......... .......... 63% 25.1M 4s 2023-12-03T11:39:25.512 INFO:teuthology.orchestra.run.smithi012.stderr: 58700K .......... .......... .......... .......... .......... 63% 9.08M 4s 2023-12-03T11:39:25.515 INFO:teuthology.orchestra.run.smithi012.stderr: 58750K .......... .......... .......... .......... .......... 63% 28.9M 4s 2023-12-03T11:39:25.516 INFO:teuthology.orchestra.run.smithi012.stderr: 58800K .......... .......... .......... .......... .......... 63% 19.3M 4s 2023-12-03T11:39:25.520 INFO:teuthology.orchestra.run.smithi012.stderr: 58850K .......... .......... .......... .......... .......... 63% 14.2M 4s 2023-12-03T11:39:25.521 INFO:teuthology.orchestra.run.smithi012.stderr: 58900K .......... .......... .......... .......... .......... 63% 33.6M 4s 2023-12-03T11:39:25.525 INFO:teuthology.orchestra.run.smithi012.stderr: 58950K .......... .......... .......... .......... .......... 63% 12.0M 4s 2023-12-03T11:39:25.528 INFO:teuthology.orchestra.run.smithi012.stderr: 59000K .......... .......... .......... .......... .......... 63% 18.6M 4s 2023-12-03T11:39:25.529 INFO:teuthology.orchestra.run.smithi012.stderr: 59050K .......... .......... .......... .......... .......... 63% 26.7M 4s 2023-12-03T11:39:25.533 INFO:teuthology.orchestra.run.smithi012.stderr: 59100K .......... .......... .......... .......... .......... 63% 14.4M 4s 2023-12-03T11:39:25.536 INFO:teuthology.orchestra.run.smithi012.stderr: 59150K .......... .......... .......... .......... .......... 63% 25.6M 4s 2023-12-03T11:39:25.538 INFO:teuthology.orchestra.run.smithi012.stderr: 59200K .......... .......... .......... .......... .......... 63% 16.1M 4s 2023-12-03T11:39:25.541 INFO:teuthology.orchestra.run.smithi012.stderr: 59250K .......... .......... .......... .......... .......... 63% 13.1M 4s 2023-12-03T11:39:25.543 INFO:teuthology.orchestra.run.smithi012.stderr: 59300K .......... .......... .......... .......... .......... 63% 27.1M 4s 2023-12-03T11:39:25.546 INFO:teuthology.orchestra.run.smithi012.stderr: 59350K .......... .......... .......... .......... .......... 63% 21.2M 4s 2023-12-03T11:39:25.549 INFO:teuthology.orchestra.run.smithi012.stderr: 59400K .......... .......... .......... .......... .......... 63% 14.5M 4s 2023-12-03T11:39:25.550 INFO:teuthology.orchestra.run.smithi012.stderr: 59450K .......... .......... .......... .......... .......... 64% 30.3M 4s 2023-12-03T11:39:25.553 INFO:teuthology.orchestra.run.smithi012.stderr: 59500K .......... .......... .......... .......... .......... 64% 24.4M 4s 2023-12-03T11:39:25.557 INFO:teuthology.orchestra.run.smithi012.stderr: 59550K .......... .......... .......... .......... .......... 64% 10.9M 4s 2023-12-03T11:39:25.560 INFO:teuthology.orchestra.run.smithi012.stderr: 59600K .......... .......... .......... .......... .......... 64% 16.7M 4s 2023-12-03T11:39:25.563 INFO:teuthology.orchestra.run.smithi012.stderr: 59650K .......... .......... .......... .......... .......... 64% 16.8M 4s 2023-12-03T11:39:25.564 INFO:teuthology.orchestra.run.smithi012.stderr: 59700K .......... .......... .......... .......... .......... 64% 35.5M 4s 2023-12-03T11:39:25.567 INFO:teuthology.orchestra.run.smithi012.stderr: 59750K .......... .......... .......... .......... .......... 64% 19.4M 4s 2023-12-03T11:39:25.571 INFO:teuthology.orchestra.run.smithi012.stderr: 59800K .......... .......... .......... .......... .......... 64% 11.6M 4s 2023-12-03T11:39:25.573 INFO:teuthology.orchestra.run.smithi012.stderr: 59850K .......... .......... .......... .......... .......... 64% 27.3M 4s 2023-12-03T11:39:25.575 INFO:teuthology.orchestra.run.smithi012.stderr: 59900K .......... .......... .......... .......... .......... 64% 24.0M 4s 2023-12-03T11:39:25.582 INFO:teuthology.orchestra.run.smithi012.stderr: 59950K .......... .......... .......... .......... .......... 64% 13.8M 4s 2023-12-03T11:39:25.582 INFO:teuthology.orchestra.run.smithi012.stderr: 60000K .......... .......... .......... .......... .......... 64% 28.3M 4s 2023-12-03T11:39:25.585 INFO:teuthology.orchestra.run.smithi012.stderr: 60050K .......... .......... .......... .......... .......... 64% 10.7M 4s 2023-12-03T11:39:25.587 INFO:teuthology.orchestra.run.smithi012.stderr: 60100K .......... .......... .......... .......... .......... 64% 24.1M 4s 2023-12-03T11:39:25.588 INFO:teuthology.orchestra.run.smithi012.stderr: 60150K .......... .......... .......... .......... .......... 64% 34.2M 3s 2023-12-03T11:39:25.593 INFO:teuthology.orchestra.run.smithi012.stderr: 60200K .......... .......... .......... .......... .......... 64% 11.1M 3s 2023-12-03T11:39:25.594 INFO:teuthology.orchestra.run.smithi012.stderr: 60250K .......... .......... .......... .......... .......... 64% 26.3M 3s 2023-12-03T11:39:25.596 INFO:teuthology.orchestra.run.smithi012.stderr: 60300K .......... .......... .......... .......... .......... 64% 34.3M 3s 2023-12-03T11:39:25.600 INFO:teuthology.orchestra.run.smithi012.stderr: 60350K .......... .......... .......... .......... .......... 65% 12.0M 3s 2023-12-03T11:39:25.602 INFO:teuthology.orchestra.run.smithi012.stderr: 60400K .......... .......... .......... .......... .......... 65% 18.9M 3s 2023-12-03T11:39:25.605 INFO:teuthology.orchestra.run.smithi012.stderr: 60450K .......... .......... .......... .......... .......... 65% 21.7M 3s 2023-12-03T11:39:25.607 INFO:teuthology.orchestra.run.smithi012.stderr: 60500K .......... .......... .......... .......... .......... 65% 18.4M 3s 2023-12-03T11:39:25.610 INFO:teuthology.orchestra.run.smithi012.stderr: 60550K .......... .......... .......... .......... .......... 65% 22.5M 3s 2023-12-03T11:39:25.612 INFO:teuthology.orchestra.run.smithi012.stderr: 60600K .......... .......... .......... .......... .......... 65% 21.6M 3s 2023-12-03T11:39:25.615 INFO:teuthology.orchestra.run.smithi012.stderr: 60650K .......... .......... .......... .......... .......... 65% 15.0M 3s 2023-12-03T11:39:25.617 INFO:teuthology.orchestra.run.smithi012.stderr: 60700K .......... .......... .......... .......... .......... 65% 19.7M 3s 2023-12-03T11:39:25.620 INFO:teuthology.orchestra.run.smithi012.stderr: 60750K .......... .......... .......... .......... .......... 65% 31.3M 3s 2023-12-03T11:39:25.623 INFO:teuthology.orchestra.run.smithi012.stderr: 60800K .......... .......... .......... .......... .......... 65% 12.5M 3s 2023-12-03T11:39:25.625 INFO:teuthology.orchestra.run.smithi012.stderr: 60850K .......... .......... .......... .......... .......... 65% 25.7M 3s 2023-12-03T11:39:25.629 INFO:teuthology.orchestra.run.smithi012.stderr: 60900K .......... .......... .......... .......... .......... 65% 11.5M 3s 2023-12-03T11:39:25.631 INFO:teuthology.orchestra.run.smithi012.stderr: 60950K .......... .......... .......... .......... .......... 65% 23.7M 3s 2023-12-03T11:39:25.636 INFO:teuthology.orchestra.run.smithi012.stderr: 61000K .......... .......... .......... .......... .......... 65% 11.3M 3s 2023-12-03T11:39:25.637 INFO:teuthology.orchestra.run.smithi012.stderr: 61050K .......... .......... .......... .......... .......... 65% 285M 3s 2023-12-03T11:39:25.639 INFO:teuthology.orchestra.run.smithi012.stderr: 61100K .......... .......... .......... .......... .......... 65% 14.0M 3s 2023-12-03T11:39:25.641 INFO:teuthology.orchestra.run.smithi012.stderr: 61150K .......... .......... .......... .......... .......... 65% 38.9M 3s 2023-12-03T11:39:25.644 INFO:teuthology.orchestra.run.smithi012.stderr: 61200K .......... .......... .......... .......... .......... 65% 13.2M 3s 2023-12-03T11:39:25.646 INFO:teuthology.orchestra.run.smithi012.stderr: 61250K .......... .......... .......... .......... .......... 65% 19.8M 3s 2023-12-03T11:39:25.651 INFO:teuthology.orchestra.run.smithi012.stderr: 61300K .......... .......... .......... .......... .......... 66% 9.89M 3s 2023-12-03T11:39:25.654 INFO:teuthology.orchestra.run.smithi012.stderr: 61350K .......... .......... .......... .......... .......... 66% 20.0M 3s 2023-12-03T11:39:25.656 INFO:teuthology.orchestra.run.smithi012.stderr: 61400K .......... .......... .......... .......... .......... 66% 25.8M 3s 2023-12-03T11:39:25.658 INFO:teuthology.orchestra.run.smithi012.stderr: 61450K .......... .......... .......... .......... .......... 66% 24.7M 3s 2023-12-03T11:39:25.661 INFO:teuthology.orchestra.run.smithi012.stderr: 61500K .......... .......... .......... .......... .......... 66% 18.1M 3s 2023-12-03T11:39:25.666 INFO:teuthology.orchestra.run.smithi012.stderr: 61550K .......... .......... .......... .......... .......... 66% 10.0M 3s 2023-12-03T11:39:25.667 INFO:teuthology.orchestra.run.smithi012.stderr: 61600K .......... .......... .......... .......... .......... 66% 22.5M 3s 2023-12-03T11:39:25.669 INFO:teuthology.orchestra.run.smithi012.stderr: 61650K .......... .......... .......... .......... .......... 66% 28.1M 3s 2023-12-03T11:39:25.672 INFO:teuthology.orchestra.run.smithi012.stderr: 61700K .......... .......... .......... .......... .......... 66% 18.5M 3s 2023-12-03T11:39:25.674 INFO:teuthology.orchestra.run.smithi012.stderr: 61750K .......... .......... .......... .......... .......... 66% 26.7M 3s 2023-12-03T11:39:25.676 INFO:teuthology.orchestra.run.smithi012.stderr: 61800K .......... .......... .......... .......... .......... 66% 22.6M 3s 2023-12-03T11:39:25.680 INFO:teuthology.orchestra.run.smithi012.stderr: 61850K .......... .......... .......... .......... .......... 66% 11.5M 3s 2023-12-03T11:39:25.683 INFO:teuthology.orchestra.run.smithi012.stderr: 61900K .......... .......... .......... .......... .......... 66% 19.7M 3s 2023-12-03T11:39:25.686 INFO:teuthology.orchestra.run.smithi012.stderr: 61950K .......... .......... .......... .......... .......... 66% 27.4M 3s 2023-12-03T11:39:25.688 INFO:teuthology.orchestra.run.smithi012.stderr: 62000K .......... .......... .......... .......... .......... 66% 16.6M 3s 2023-12-03T11:39:25.692 INFO:teuthology.orchestra.run.smithi012.stderr: 62050K .......... .......... .......... .......... .......... 66% 21.5M 3s 2023-12-03T11:39:25.692 INFO:teuthology.orchestra.run.smithi012.stderr: 62100K .......... .......... .......... .......... .......... 66% 39.7M 3s 2023-12-03T11:39:25.697 INFO:teuthology.orchestra.run.smithi012.stderr: 62150K .......... .......... .......... .......... .......... 66% 8.91M 3s 2023-12-03T11:39:25.698 INFO:teuthology.orchestra.run.smithi012.stderr: 62200K .......... .......... .......... .......... .......... 67% 30.1M 3s 2023-12-03T11:39:25.703 INFO:teuthology.orchestra.run.smithi012.stderr: 62250K .......... .......... .......... .......... .......... 67% 9.83M 3s 2023-12-03T11:39:25.705 INFO:teuthology.orchestra.run.smithi012.stderr: 62300K .......... .......... .......... .......... .......... 67% 30.3M 3s 2023-12-03T11:39:25.707 INFO:teuthology.orchestra.run.smithi012.stderr: 62350K .......... .......... .......... .......... .......... 67% 18.6M 3s 2023-12-03T11:39:25.710 INFO:teuthology.orchestra.run.smithi012.stderr: 62400K .......... .......... .......... .......... .......... 67% 18.0M 3s 2023-12-03T11:39:25.712 INFO:teuthology.orchestra.run.smithi012.stderr: 62450K .......... .......... .......... .......... .......... 67% 26.5M 3s 2023-12-03T11:39:25.717 INFO:teuthology.orchestra.run.smithi012.stderr: 62500K .......... .......... .......... .......... .......... 67% 9.50M 3s 2023-12-03T11:39:25.718 INFO:teuthology.orchestra.run.smithi012.stderr: 62550K .......... .......... .......... .......... .......... 67% 38.2M 3s 2023-12-03T11:39:25.721 INFO:teuthology.orchestra.run.smithi012.stderr: 62600K .......... .......... .......... .......... .......... 67% 19.1M 3s 2023-12-03T11:39:25.722 INFO:teuthology.orchestra.run.smithi012.stderr: 62650K .......... .......... .......... .......... .......... 67% 25.5M 3s 2023-12-03T11:39:25.724 INFO:teuthology.orchestra.run.smithi012.stderr: 62700K .......... .......... .......... .......... .......... 67% 28.0M 3s 2023-12-03T11:39:25.727 INFO:teuthology.orchestra.run.smithi012.stderr: 62750K .......... .......... .......... .......... .......... 67% 28.2M 3s 2023-12-03T11:39:25.733 INFO:teuthology.orchestra.run.smithi012.stderr: 62800K .......... .......... .......... .......... .......... 67% 7.92M 3s 2023-12-03T11:39:25.734 INFO:teuthology.orchestra.run.smithi012.stderr: 62850K .......... .......... .......... .......... .......... 67% 26.1M 3s 2023-12-03T11:39:25.736 INFO:teuthology.orchestra.run.smithi012.stderr: 62900K .......... .......... .......... .......... .......... 67% 26.6M 3s 2023-12-03T11:39:25.738 INFO:teuthology.orchestra.run.smithi012.stderr: 62950K .......... .......... .......... .......... .......... 67% 28.1M 3s 2023-12-03T11:39:25.741 INFO:teuthology.orchestra.run.smithi012.stderr: 63000K .......... .......... .......... .......... .......... 67% 18.2M 3s 2023-12-03T11:39:25.742 INFO:teuthology.orchestra.run.smithi012.stderr: 63050K .......... .......... .......... .......... .......... 67% 34.7M 3s 2023-12-03T11:39:25.746 INFO:teuthology.orchestra.run.smithi012.stderr: 63100K .......... .......... .......... .......... .......... 67% 10.5M 3s 2023-12-03T11:39:25.749 INFO:teuthology.orchestra.run.smithi012.stderr: 63150K .......... .......... .......... .......... .......... 68% 27.9M 3s 2023-12-03T11:39:25.752 INFO:teuthology.orchestra.run.smithi012.stderr: 63200K .......... .......... .......... .......... .......... 68% 16.5M 3s 2023-12-03T11:39:25.754 INFO:teuthology.orchestra.run.smithi012.stderr: 63250K .......... .......... .......... .......... .......... 68% 23.8M 3s 2023-12-03T11:39:25.755 INFO:teuthology.orchestra.run.smithi012.stderr: 63300K .......... .......... .......... .......... .......... 68% 25.0M 3s 2023-12-03T11:39:25.757 INFO:teuthology.orchestra.run.smithi012.stderr: 63350K .......... .......... .......... .......... .......... 68% 29.7M 3s 2023-12-03T11:39:25.762 INFO:teuthology.orchestra.run.smithi012.stderr: 63400K .......... .......... .......... .......... .......... 68% 8.88M 3s 2023-12-03T11:39:25.764 INFO:teuthology.orchestra.run.smithi012.stderr: 63450K .......... .......... .......... .......... .......... 68% 26.0M 3s 2023-12-03T11:39:25.766 INFO:teuthology.orchestra.run.smithi012.stderr: 63500K .......... .......... .......... .......... .......... 68% 27.0M 3s 2023-12-03T11:39:25.768 INFO:teuthology.orchestra.run.smithi012.stderr: 63550K .......... .......... .......... .......... .......... 68% 30.2M 3s 2023-12-03T11:39:25.770 INFO:teuthology.orchestra.run.smithi012.stderr: 63600K .......... .......... .......... .......... .......... 68% 18.7M 3s 2023-12-03T11:39:25.772 INFO:teuthology.orchestra.run.smithi012.stderr: 63650K .......... .......... .......... .......... .......... 68% 28.1M 3s 2023-12-03T11:39:25.777 INFO:teuthology.orchestra.run.smithi012.stderr: 63700K .......... .......... .......... .......... .......... 68% 9.09M 3s 2023-12-03T11:39:25.779 INFO:teuthology.orchestra.run.smithi012.stderr: 63750K .......... .......... .......... .......... .......... 68% 27.2M 3s 2023-12-03T11:39:25.781 INFO:teuthology.orchestra.run.smithi012.stderr: 63800K .......... .......... .......... .......... .......... 68% 27.4M 3s 2023-12-03T11:39:25.783 INFO:teuthology.orchestra.run.smithi012.stderr: 63850K .......... .......... .......... .......... .......... 68% 27.9M 3s 2023-12-03T11:39:25.785 INFO:teuthology.orchestra.run.smithi012.stderr: 63900K .......... .......... .......... .......... .......... 68% 19.2M 3s 2023-12-03T11:39:25.788 INFO:teuthology.orchestra.run.smithi012.stderr: 63950K .......... .......... .......... .......... .......... 68% 29.0M 3s 2023-12-03T11:39:25.792 INFO:teuthology.orchestra.run.smithi012.stderr: 64000K .......... .......... .......... .......... .......... 68% 8.78M 3s 2023-12-03T11:39:25.794 INFO:teuthology.orchestra.run.smithi012.stderr: 64050K .......... .......... .......... .......... .......... 68% 32.6M 3s 2023-12-03T11:39:25.796 INFO:teuthology.orchestra.run.smithi012.stderr: 64100K .......... .......... .......... .......... .......... 69% 19.6M 3s 2023-12-03T11:39:25.798 INFO:teuthology.orchestra.run.smithi012.stderr: 64150K .......... .......... .......... .......... .......... 69% 28.0M 3s 2023-12-03T11:39:25.800 INFO:teuthology.orchestra.run.smithi012.stderr: 64200K .......... .......... .......... .......... .......... 69% 25.4M 3s 2023-12-03T11:39:25.802 INFO:teuthology.orchestra.run.smithi012.stderr: 64250K .......... .......... .......... .......... .......... 69% 30.1M 3s 2023-12-03T11:39:25.807 INFO:teuthology.orchestra.run.smithi012.stderr: 64300K .......... .......... .......... .......... .......... 69% 8.54M 3s 2023-12-03T11:39:25.810 INFO:teuthology.orchestra.run.smithi012.stderr: 64350K .......... .......... .......... .......... .......... 69% 29.4M 3s 2023-12-03T11:39:25.811 INFO:teuthology.orchestra.run.smithi012.stderr: 64400K .......... .......... .......... .......... .......... 69% 19.0M 3s 2023-12-03T11:39:25.813 INFO:teuthology.orchestra.run.smithi012.stderr: 64450K .......... .......... .......... .......... .......... 69% 28.2M 3s 2023-12-03T11:39:25.816 INFO:teuthology.orchestra.run.smithi012.stderr: 64500K .......... .......... .......... .......... .......... 69% 19.1M 3s 2023-12-03T11:39:25.818 INFO:teuthology.orchestra.run.smithi012.stderr: 64550K .......... .......... .......... .......... .......... 69% 26.1M 3s 2023-12-03T11:39:25.823 INFO:teuthology.orchestra.run.smithi012.stderr: 64600K .......... .......... .......... .......... .......... 69% 10.6M 3s 2023-12-03T11:39:25.824 INFO:teuthology.orchestra.run.smithi012.stderr: 64650K .......... .......... .......... .......... .......... 69% 32.0M 3s 2023-12-03T11:39:25.827 INFO:teuthology.orchestra.run.smithi012.stderr: 64700K .......... .......... .......... .......... .......... 69% 18.9M 3s 2023-12-03T11:39:25.829 INFO:teuthology.orchestra.run.smithi012.stderr: 64750K .......... .......... .......... .......... .......... 69% 28.4M 3s 2023-12-03T11:39:25.832 INFO:teuthology.orchestra.run.smithi012.stderr: 64800K .......... .......... .......... .......... .......... 69% 14.6M 3s 2023-12-03T11:39:25.833 INFO:teuthology.orchestra.run.smithi012.stderr: 64850K .......... .......... .......... .......... .......... 69% 35.6M 3s 2023-12-03T11:39:25.837 INFO:teuthology.orchestra.run.smithi012.stderr: 64900K .......... .......... .......... .......... .......... 69% 10.6M 3s 2023-12-03T11:39:25.840 INFO:teuthology.orchestra.run.smithi012.stderr: 64950K .......... .......... .......... .......... .......... 69% 22.9M 3s 2023-12-03T11:39:25.842 INFO:teuthology.orchestra.run.smithi012.stderr: 65000K .......... .......... .......... .......... .......... 70% 23.1M 3s 2023-12-03T11:39:25.843 INFO:teuthology.orchestra.run.smithi012.stderr: 65050K .......... .......... .......... .......... .......... 70% 30.4M 3s 2023-12-03T11:39:25.845 INFO:teuthology.orchestra.run.smithi012.stderr: 65100K .......... .......... .......... .......... .......... 70% 28.6M 3s 2023-12-03T11:39:25.848 INFO:teuthology.orchestra.run.smithi012.stderr: 65150K .......... .......... .......... .......... .......... 70% 17.9M 3s 2023-12-03T11:39:25.853 INFO:teuthology.orchestra.run.smithi012.stderr: 65200K .......... .......... .......... .......... .......... 70% 8.96M 3s 2023-12-03T11:39:25.855 INFO:teuthology.orchestra.run.smithi012.stderr: 65250K .......... .......... .......... .......... .......... 70% 34.0M 3s 2023-12-03T11:39:25.857 INFO:teuthology.orchestra.run.smithi012.stderr: 65300K .......... .......... .......... .......... .......... 70% 23.4M 3s 2023-12-03T11:39:25.858 INFO:teuthology.orchestra.run.smithi012.stderr: 65350K .......... .......... .......... .......... .......... 70% 35.7M 3s 2023-12-03T11:39:25.862 INFO:teuthology.orchestra.run.smithi012.stderr: 65400K .......... .......... .......... .......... .......... 70% 15.4M 3s 2023-12-03T11:39:25.863 INFO:teuthology.orchestra.run.smithi012.stderr: 65450K .......... .......... .......... .......... .......... 70% 34.0M 3s 2023-12-03T11:39:25.867 INFO:teuthology.orchestra.run.smithi012.stderr: 65500K .......... .......... .......... .......... .......... 70% 11.0M 3s 2023-12-03T11:39:25.870 INFO:teuthology.orchestra.run.smithi012.stderr: 65550K .......... .......... .......... .......... .......... 70% 20.7M 3s 2023-12-03T11:39:25.872 INFO:teuthology.orchestra.run.smithi012.stderr: 65600K .......... .......... .......... .......... .......... 70% 23.2M 3s 2023-12-03T11:39:25.874 INFO:teuthology.orchestra.run.smithi012.stderr: 65650K .......... .......... .......... .......... .......... 70% 23.1M 3s 2023-12-03T11:39:25.876 INFO:teuthology.orchestra.run.smithi012.stderr: 65700K .......... .......... .......... .......... .......... 70% 21.5M 3s 2023-12-03T11:39:25.878 INFO:teuthology.orchestra.run.smithi012.stderr: 65750K .......... .......... .......... .......... .......... 70% 25.7M 3s 2023-12-03T11:39:25.883 INFO:teuthology.orchestra.run.smithi012.stderr: 65800K .......... .......... .......... .......... .......... 70% 9.69M 3s 2023-12-03T11:39:25.884 INFO:teuthology.orchestra.run.smithi012.stderr: 65850K .......... .......... .......... .......... .......... 70% 33.2M 3s 2023-12-03T11:39:25.886 INFO:teuthology.orchestra.run.smithi012.stderr: 65900K .......... .......... .......... .......... .......... 70% 23.1M 3s 2023-12-03T11:39:25.889 INFO:teuthology.orchestra.run.smithi012.stderr: 65950K .......... .......... .......... .......... .......... 71% 35.2M 3s 2023-12-03T11:39:25.891 INFO:teuthology.orchestra.run.smithi012.stderr: 66000K .......... .......... .......... .......... .......... 71% 16.4M 3s 2023-12-03T11:39:25.896 INFO:teuthology.orchestra.run.smithi012.stderr: 66050K .......... .......... .......... .......... .......... 71% 12.3M 3s 2023-12-03T11:39:25.900 INFO:teuthology.orchestra.run.smithi012.stderr: 66100K .......... .......... .......... .......... .......... 71% 16.4M 3s 2023-12-03T11:39:25.900 INFO:teuthology.orchestra.run.smithi012.stderr: 66150K .......... .......... .......... .......... .......... 71% 34.3M 3s 2023-12-03T11:39:25.902 INFO:teuthology.orchestra.run.smithi012.stderr: 66200K .......... .......... .......... .......... .......... 71% 20.3M 3s 2023-12-03T11:39:25.904 INFO:teuthology.orchestra.run.smithi012.stderr: 66250K .......... .......... .......... .......... .......... 71% 22.9M 3s 2023-12-03T11:39:25.906 INFO:teuthology.orchestra.run.smithi012.stderr: 66300K .......... .......... .......... .......... .......... 71% 24.2M 3s 2023-12-03T11:39:25.910 INFO:teuthology.orchestra.run.smithi012.stderr: 66350K .......... .......... .......... .......... .......... 71% 26.1M 3s 2023-12-03T11:39:25.913 INFO:teuthology.orchestra.run.smithi012.stderr: 66400K .......... .......... .......... .......... .......... 71% 9.72M 3s 2023-12-03T11:39:25.915 INFO:teuthology.orchestra.run.smithi012.stderr: 66450K .......... .......... .......... .......... .......... 71% 30.9M 3s 2023-12-03T11:39:25.917 INFO:teuthology.orchestra.run.smithi012.stderr: 66500K .......... .......... .......... .......... .......... 71% 19.6M 3s 2023-12-03T11:39:25.918 INFO:teuthology.orchestra.run.smithi012.stderr: 66550K .......... .......... .......... .......... .......... 71% 28.4M 3s 2023-12-03T11:39:25.921 INFO:teuthology.orchestra.run.smithi012.stderr: 66600K .......... .......... .......... .......... .......... 71% 21.6M 3s 2023-12-03T11:39:25.923 INFO:teuthology.orchestra.run.smithi012.stderr: 66650K .......... .......... .......... .......... .......... 71% 30.2M 3s 2023-12-03T11:39:25.927 INFO:teuthology.orchestra.run.smithi012.stderr: 66700K .......... .......... .......... .......... .......... 71% 9.55M 3s 2023-12-03T11:39:25.930 INFO:teuthology.orchestra.run.smithi012.stderr: 66750K .......... .......... .......... .......... .......... 71% 31.4M 3s 2023-12-03T11:39:25.932 INFO:teuthology.orchestra.run.smithi012.stderr: 66800K .......... .......... .......... .......... .......... 71% 17.9M 3s 2023-12-03T11:39:25.934 INFO:teuthology.orchestra.run.smithi012.stderr: 66850K .......... .......... .......... .......... .......... 72% 28.4M 3s 2023-12-03T11:39:25.936 INFO:teuthology.orchestra.run.smithi012.stderr: 66900K .......... .......... .......... .......... .......... 72% 18.9M 3s 2023-12-03T11:39:25.939 INFO:teuthology.orchestra.run.smithi012.stderr: 66950K .......... .......... .......... .......... .......... 72% 15.5M 3s 2023-12-03T11:39:25.942 INFO:teuthology.orchestra.run.smithi012.stderr: 67000K .......... .......... .......... .......... .......... 72% 16.6M 3s 2023-12-03T11:39:25.945 INFO:teuthology.orchestra.run.smithi012.stderr: 67050K .......... .......... .......... .......... .......... 72% 18.1M 3s 2023-12-03T11:39:25.947 INFO:teuthology.orchestra.run.smithi012.stderr: 67100K .......... .......... .......... .......... .......... 72% 28.7M 3s 2023-12-03T11:39:25.949 INFO:teuthology.orchestra.run.smithi012.stderr: 67150K .......... .......... .......... .......... .......... 72% 29.1M 3s 2023-12-03T11:39:25.951 INFO:teuthology.orchestra.run.smithi012.stderr: 67200K .......... .......... .......... .......... .......... 72% 18.2M 3s 2023-12-03T11:39:25.954 INFO:teuthology.orchestra.run.smithi012.stderr: 67250K .......... .......... .......... .......... .......... 72% 16.2M 3s 2023-12-03T11:39:25.958 INFO:teuthology.orchestra.run.smithi012.stderr: 67300K .......... .......... .......... .......... .......... 72% 12.2M 3s 2023-12-03T11:39:25.960 INFO:teuthology.orchestra.run.smithi012.stderr: 67350K .......... .......... .......... .......... .......... 72% 25.7M 3s 2023-12-03T11:39:25.961 INFO:teuthology.orchestra.run.smithi012.stderr: 67400K .......... .......... .......... .......... .......... 72% 29.3M 3s 2023-12-03T11:39:25.963 INFO:teuthology.orchestra.run.smithi012.stderr: 67450K .......... .......... .......... .......... .......... 72% 27.2M 3s 2023-12-03T11:39:25.966 INFO:teuthology.orchestra.run.smithi012.stderr: 67500K .......... .......... .......... .......... .......... 72% 20.0M 3s 2023-12-03T11:39:25.969 INFO:teuthology.orchestra.run.smithi012.stderr: 67550K .......... .......... .......... .......... .......... 72% 16.6M 3s 2023-12-03T11:39:25.973 INFO:teuthology.orchestra.run.smithi012.stderr: 67600K .......... .......... .......... .......... .......... 72% 11.9M 3s 2023-12-03T11:39:25.975 INFO:teuthology.orchestra.run.smithi012.stderr: 67650K .......... .......... .......... .......... .......... 72% 25.9M 3s 2023-12-03T11:39:25.976 INFO:teuthology.orchestra.run.smithi012.stderr: 67700K .......... .......... .......... .......... .......... 72% 29.0M 3s 2023-12-03T11:39:25.979 INFO:teuthology.orchestra.run.smithi012.stderr: 67750K .......... .......... .......... .......... .......... 72% 20.5M 3s 2023-12-03T11:39:25.981 INFO:teuthology.orchestra.run.smithi012.stderr: 67800K .......... .......... .......... .......... .......... 73% 25.1M 3s 2023-12-03T11:39:25.983 INFO:teuthology.orchestra.run.smithi012.stderr: 67850K .......... .......... .......... .......... .......... 73% 17.3M 3s 2023-12-03T11:39:25.990 INFO:teuthology.orchestra.run.smithi012.stderr: 67900K .......... .......... .......... .......... .......... 73% 15.4M 3s 2023-12-03T11:39:25.990 INFO:teuthology.orchestra.run.smithi012.stderr: 67950K .......... .......... .......... .......... .......... 73% 16.9M 3s 2023-12-03T11:39:25.992 INFO:teuthology.orchestra.run.smithi012.stderr: 68000K .......... .......... .......... .......... .......... 73% 21.3M 3s 2023-12-03T11:39:25.994 INFO:teuthology.orchestra.run.smithi012.stderr: 68050K .......... .......... .......... .......... .......... 73% 27.1M 2s 2023-12-03T11:39:25.995 INFO:teuthology.orchestra.run.smithi012.stderr: 68100K .......... .......... .......... .......... .......... 73% 26.4M 2s 2023-12-03T11:39:25.999 INFO:teuthology.orchestra.run.smithi012.stderr: 68150K .......... .......... .......... .......... .......... 73% 12.7M 2s 2023-12-03T11:39:26.002 INFO:teuthology.orchestra.run.smithi012.stderr: 68200K .......... .......... .......... .......... .......... 73% 17.8M 2s 2023-12-03T11:39:26.004 INFO:teuthology.orchestra.run.smithi012.stderr: 68250K .......... .......... .......... .......... .......... 73% 26.7M 2s 2023-12-03T11:39:26.006 INFO:teuthology.orchestra.run.smithi012.stderr: 68300K .......... .......... .......... .......... .......... 73% 29.1M 2s 2023-12-03T11:39:26.008 INFO:teuthology.orchestra.run.smithi012.stderr: 68350K .......... .......... .......... .......... .......... 73% 19.8M 2s 2023-12-03T11:39:26.011 INFO:teuthology.orchestra.run.smithi012.stderr: 68400K .......... .......... .......... .......... .......... 73% 19.6M 2s 2023-12-03T11:39:26.017 INFO:teuthology.orchestra.run.smithi012.stderr: 68450K .......... .......... .......... .......... .......... 73% 8.95M 2s 2023-12-03T11:39:26.017 INFO:teuthology.orchestra.run.smithi012.stderr: 68500K .......... .......... .......... .......... .......... 73% 37.3M 2s 2023-12-03T11:39:26.019 INFO:teuthology.orchestra.run.smithi012.stderr: 68550K .......... .......... .......... .......... .......... 73% 32.7M 2s 2023-12-03T11:39:26.022 INFO:teuthology.orchestra.run.smithi012.stderr: 68600K .......... .......... .......... .......... .......... 73% 18.7M 2s 2023-12-03T11:39:26.023 INFO:teuthology.orchestra.run.smithi012.stderr: 68650K .......... .......... .......... .......... .......... 73% 29.3M 2s 2023-12-03T11:39:26.025 INFO:teuthology.orchestra.run.smithi012.stderr: 68700K .......... .......... .......... .......... .......... 74% 26.9M 2s 2023-12-03T11:39:26.031 INFO:teuthology.orchestra.run.smithi012.stderr: 68750K .......... .......... .......... .......... .......... 74% 31.3M 2s 2023-12-03T11:39:26.033 INFO:teuthology.orchestra.run.smithi012.stderr: 68800K .......... .......... .......... .......... .......... 74% 8.10M 2s 2023-12-03T11:39:26.035 INFO:teuthology.orchestra.run.smithi012.stderr: 68850K .......... .......... .......... .......... .......... 74% 26.3M 2s 2023-12-03T11:39:26.037 INFO:teuthology.orchestra.run.smithi012.stderr: 68900K .......... .......... .......... .......... .......... 74% 25.0M 2s 2023-12-03T11:39:26.039 INFO:teuthology.orchestra.run.smithi012.stderr: 68950K .......... .......... .......... .......... .......... 74% 19.9M 2s 2023-12-03T11:39:26.040 INFO:teuthology.orchestra.run.smithi012.stderr: 69000K .......... .......... .......... .......... .......... 74% 31.4M 2s 2023-12-03T11:39:26.044 INFO:teuthology.orchestra.run.smithi012.stderr: 69050K .......... .......... .......... .......... .......... 74% 15.4M 2s 2023-12-03T11:39:26.048 INFO:teuthology.orchestra.run.smithi012.stderr: 69100K .......... .......... .......... .......... .......... 74% 11.5M 2s 2023-12-03T11:39:26.050 INFO:teuthology.orchestra.run.smithi012.stderr: 69150K .......... .......... .......... .......... .......... 74% 19.2M 2s 2023-12-03T11:39:26.053 INFO:teuthology.orchestra.run.smithi012.stderr: 69200K .......... .......... .......... .......... .......... 74% 20.1M 2s 2023-12-03T11:39:26.055 INFO:teuthology.orchestra.run.smithi012.stderr: 69250K .......... .......... .......... .......... .......... 74% 25.3M 2s 2023-12-03T11:39:26.056 INFO:teuthology.orchestra.run.smithi012.stderr: 69300K .......... .......... .......... .......... .......... 74% 28.6M 2s 2023-12-03T11:39:26.058 INFO:teuthology.orchestra.run.smithi012.stderr: 69350K .......... .......... .......... .......... .......... 74% 24.7M 2s 2023-12-03T11:39:26.064 INFO:teuthology.orchestra.run.smithi012.stderr: 69400K .......... .......... .......... .......... .......... 74% 9.25M 2s 2023-12-03T11:39:26.065 INFO:teuthology.orchestra.run.smithi012.stderr: 69450K .......... .......... .......... .......... .......... 74% 33.6M 2s 2023-12-03T11:39:26.067 INFO:teuthology.orchestra.run.smithi012.stderr: 69500K .......... .......... .......... .......... .......... 74% 25.8M 2s 2023-12-03T11:39:26.070 INFO:teuthology.orchestra.run.smithi012.stderr: 69550K .......... .......... .......... .......... .......... 74% 26.0M 2s 2023-12-03T11:39:26.071 INFO:teuthology.orchestra.run.smithi012.stderr: 69600K .......... .......... .......... .......... .......... 74% 21.4M 2s 2023-12-03T11:39:26.077 INFO:teuthology.orchestra.run.smithi012.stderr: 69650K .......... .......... .......... .......... .......... 75% 8.29M 2s 2023-12-03T11:39:26.078 INFO:teuthology.orchestra.run.smithi012.stderr: 69700K .......... .......... .......... .......... .......... 75% 34.8M 2s 2023-12-03T11:39:26.080 INFO:teuthology.orchestra.run.smithi012.stderr: 69750K .......... .......... .......... .......... .......... 75% 29.6M 2s 2023-12-03T11:39:26.082 INFO:teuthology.orchestra.run.smithi012.stderr: 69800K .......... .......... .......... .......... .......... 75% 29.9M 2s 2023-12-03T11:39:26.084 INFO:teuthology.orchestra.run.smithi012.stderr: 69850K .......... .......... .......... .......... .......... 75% 20.5M 2s 2023-12-03T11:39:26.086 INFO:teuthology.orchestra.run.smithi012.stderr: 69900K .......... .......... .......... .......... .......... 75% 26.0M 2s 2023-12-03T11:39:26.088 INFO:teuthology.orchestra.run.smithi012.stderr: 69950K .......... .......... .......... .......... .......... 75% 30.3M 2s 2023-12-03T11:39:26.093 INFO:teuthology.orchestra.run.smithi012.stderr: 70000K .......... .......... .......... .......... .......... 75% 8.48M 2s 2023-12-03T11:39:26.096 INFO:teuthology.orchestra.run.smithi012.stderr: 70050K .......... .......... .......... .......... .......... 75% 22.6M 2s 2023-12-03T11:39:26.098 INFO:teuthology.orchestra.run.smithi012.stderr: 70100K .......... .......... .......... .......... .......... 75% 25.6M 2s 2023-12-03T11:39:26.099 INFO:teuthology.orchestra.run.smithi012.stderr: 70150K .......... .......... .......... .......... .......... 75% 25.7M 2s 2023-12-03T11:39:26.101 INFO:teuthology.orchestra.run.smithi012.stderr: 70200K .......... .......... .......... .......... .......... 75% 29.9M 2s 2023-12-03T11:39:26.104 INFO:teuthology.orchestra.run.smithi012.stderr: 70250K .......... .......... .......... .......... .......... 75% 19.8M 2s 2023-12-03T11:39:26.107 INFO:teuthology.orchestra.run.smithi012.stderr: 70300K .......... .......... .......... .......... .......... 75% 12.3M 2s 2023-12-03T11:39:26.110 INFO:teuthology.orchestra.run.smithi012.stderr: 70350K .......... .......... .......... .......... .......... 75% 32.2M 2s 2023-12-03T11:39:26.112 INFO:teuthology.orchestra.run.smithi012.stderr: 70400K .......... .......... .......... .......... .......... 75% 16.4M 2s 2023-12-03T11:39:26.113 INFO:teuthology.orchestra.run.smithi012.stderr: 70450K .......... .......... .......... .......... .......... 75% 30.2M 2s 2023-12-03T11:39:26.116 INFO:teuthology.orchestra.run.smithi012.stderr: 70500K .......... .......... .......... .......... .......... 75% 20.7M 2s 2023-12-03T11:39:26.117 INFO:teuthology.orchestra.run.smithi012.stderr: 70550K .......... .......... .......... .......... .......... 75% 31.0M 2s 2023-12-03T11:39:26.121 INFO:teuthology.orchestra.run.smithi012.stderr: 70600K .......... .......... .......... .......... .......... 76% 11.8M 2s 2023-12-03T11:39:26.123 INFO:teuthology.orchestra.run.smithi012.stderr: 70650K .......... .......... .......... .......... .......... 76% 26.2M 2s 2023-12-03T11:39:26.126 INFO:teuthology.orchestra.run.smithi012.stderr: 70700K .......... .......... .......... .......... .......... 76% 19.1M 2s 2023-12-03T11:39:26.129 INFO:teuthology.orchestra.run.smithi012.stderr: 70750K .......... .......... .......... .......... .......... 76% 29.8M 2s 2023-12-03T11:39:26.130 INFO:teuthology.orchestra.run.smithi012.stderr: 70800K .......... .......... .......... .......... .......... 76% 19.8M 2s 2023-12-03T11:39:26.132 INFO:teuthology.orchestra.run.smithi012.stderr: 70850K .......... .......... .......... .......... .......... 76% 25.9M 2s 2023-12-03T11:39:26.134 INFO:teuthology.orchestra.run.smithi012.stderr: 70900K .......... .......... .......... .......... .......... 76% 30.4M 2s 2023-12-03T11:39:26.139 INFO:teuthology.orchestra.run.smithi012.stderr: 70950K .......... .......... .......... .......... .......... 76% 10.0M 2s 2023-12-03T11:39:26.141 INFO:teuthology.orchestra.run.smithi012.stderr: 71000K .......... .......... .......... .......... .......... 76% 21.2M 2s 2023-12-03T11:39:26.142 INFO:teuthology.orchestra.run.smithi012.stderr: 71050K .......... .......... .......... .......... .......... 76% 37.1M 2s 2023-12-03T11:39:26.144 INFO:teuthology.orchestra.run.smithi012.stderr: 71100K .......... .......... .......... .......... .......... 76% 28.5M 2s 2023-12-03T11:39:26.146 INFO:teuthology.orchestra.run.smithi012.stderr: 71150K .......... .......... .......... .......... .......... 76% 20.8M 2s 2023-12-03T11:39:26.149 INFO:teuthology.orchestra.run.smithi012.stderr: 71200K .......... .......... .......... .......... .......... 76% 19.8M 2s 2023-12-03T11:39:26.153 INFO:teuthology.orchestra.run.smithi012.stderr: 71250K .......... .......... .......... .......... .......... 76% 12.4M 2s 2023-12-03T11:39:26.154 INFO:teuthology.orchestra.run.smithi012.stderr: 71300K .......... .......... .......... .......... .......... 76% 32.0M 2s 2023-12-03T11:39:26.157 INFO:teuthology.orchestra.run.smithi012.stderr: 71350K .......... .......... .......... .......... .......... 76% 18.2M 2s 2023-12-03T11:39:26.159 INFO:teuthology.orchestra.run.smithi012.stderr: 71400K .......... .......... .......... .......... .......... 76% 24.2M 2s 2023-12-03T11:39:26.160 INFO:teuthology.orchestra.run.smithi012.stderr: 71450K .......... .......... .......... .......... .......... 76% 30.8M 2s 2023-12-03T11:39:26.163 INFO:teuthology.orchestra.run.smithi012.stderr: 71500K .......... .......... .......... .......... .......... 77% 20.4M 2s 2023-12-03T11:39:26.168 INFO:teuthology.orchestra.run.smithi012.stderr: 71550K .......... .......... .......... .......... .......... 77% 44.4M 2s 2023-12-03T11:39:26.169 INFO:teuthology.orchestra.run.smithi012.stderr: 71600K .......... .......... .......... .......... .......... 77% 9.33M 2s 2023-12-03T11:39:26.172 INFO:teuthology.orchestra.run.smithi012.stderr: 71650K .......... .......... .......... .......... .......... 77% 19.0M 2s 2023-12-03T11:39:26.173 INFO:teuthology.orchestra.run.smithi012.stderr: 71700K .......... .......... .......... .......... .......... 77% 32.8M 2s 2023-12-03T11:39:26.176 INFO:teuthology.orchestra.run.smithi012.stderr: 71750K .......... .......... .......... .......... .......... 77% 20.7M 2s 2023-12-03T11:39:26.177 INFO:teuthology.orchestra.run.smithi012.stderr: 71800K .......... .......... .......... .......... .......... 77% 26.1M 2s 2023-12-03T11:39:26.179 INFO:teuthology.orchestra.run.smithi012.stderr: 71850K .......... .......... .......... .......... .......... 77% 30.4M 2s 2023-12-03T11:39:26.183 INFO:teuthology.orchestra.run.smithi012.stderr: 71900K .......... .......... .......... .......... .......... 77% 13.4M 2s 2023-12-03T11:39:26.187 INFO:teuthology.orchestra.run.smithi012.stderr: 71950K .......... .......... .......... .......... .......... 77% 15.3M 2s 2023-12-03T11:39:26.188 INFO:teuthology.orchestra.run.smithi012.stderr: 72000K .......... .......... .......... .......... .......... 77% 22.6M 2s 2023-12-03T11:39:26.190 INFO:teuthology.orchestra.run.smithi012.stderr: 72050K .......... .......... .......... .......... .......... 77% 25.7M 2s 2023-12-03T11:39:26.194 INFO:teuthology.orchestra.run.smithi012.stderr: 72100K .......... .......... .......... .......... .......... 77% 28.8M 2s 2023-12-03T11:39:26.194 INFO:teuthology.orchestra.run.smithi012.stderr: 72150K .......... .......... .......... .......... .......... 77% 26.8M 2s 2023-12-03T11:39:26.198 INFO:teuthology.orchestra.run.smithi012.stderr: 72200K .......... .......... .......... .......... .......... 77% 11.7M 2s 2023-12-03T11:39:26.199 INFO:teuthology.orchestra.run.smithi012.stderr: 72250K .......... .......... .......... .......... .......... 77% 33.0M 2s 2023-12-03T11:39:26.203 INFO:teuthology.orchestra.run.smithi012.stderr: 72300K .......... .......... .......... .......... .......... 77% 11.7M 2s 2023-12-03T11:39:26.205 INFO:teuthology.orchestra.run.smithi012.stderr: 72350K .......... .......... .......... .......... .......... 77% 44.1M 2s 2023-12-03T11:39:26.207 INFO:teuthology.orchestra.run.smithi012.stderr: 72400K .......... .......... .......... .......... .......... 77% 19.8M 2s 2023-12-03T11:39:26.209 INFO:teuthology.orchestra.run.smithi012.stderr: 72450K .......... .......... .......... .......... .......... 78% 20.9M 2s 2023-12-03T11:39:26.211 INFO:teuthology.orchestra.run.smithi012.stderr: 72500K .......... .......... .......... .......... .......... 78% 25.0M 2s 2023-12-03T11:39:26.213 INFO:teuthology.orchestra.run.smithi012.stderr: 72550K .......... .......... .......... .......... .......... 78% 19.2M 2s 2023-12-03T11:39:26.218 INFO:teuthology.orchestra.run.smithi012.stderr: 72600K .......... .......... .......... .......... .......... 78% 11.5M 2s 2023-12-03T11:39:26.219 INFO:teuthology.orchestra.run.smithi012.stderr: 72650K .......... .......... .......... .......... .......... 78% 32.6M 2s 2023-12-03T11:39:26.221 INFO:teuthology.orchestra.run.smithi012.stderr: 72700K .......... .......... .......... .......... .......... 78% 26.1M 2s 2023-12-03T11:39:26.224 INFO:teuthology.orchestra.run.smithi012.stderr: 72750K .......... .......... .......... .......... .......... 78% 30.2M 2s 2023-12-03T11:39:26.225 INFO:teuthology.orchestra.run.smithi012.stderr: 72800K .......... .......... .......... .......... .......... 78% 20.0M 2s 2023-12-03T11:39:26.229 INFO:teuthology.orchestra.run.smithi012.stderr: 72850K .......... .......... .......... .......... .......... 78% 14.8M 2s 2023-12-03T11:39:26.231 INFO:teuthology.orchestra.run.smithi012.stderr: 72900K .......... .......... .......... .......... .......... 78% 19.2M 2s 2023-12-03T11:39:26.234 INFO:teuthology.orchestra.run.smithi012.stderr: 72950K .......... .......... .......... .......... .......... 78% 17.5M 2s 2023-12-03T11:39:26.236 INFO:teuthology.orchestra.run.smithi012.stderr: 73000K .......... .......... .......... .......... .......... 78% 29.3M 2s 2023-12-03T11:39:26.238 INFO:teuthology.orchestra.run.smithi012.stderr: 73050K .......... .......... .......... .......... .......... 78% 20.3M 2s 2023-12-03T11:39:26.239 INFO:teuthology.orchestra.run.smithi012.stderr: 73100K .......... .......... .......... .......... .......... 78% 32.5M 2s 2023-12-03T11:39:26.242 INFO:teuthology.orchestra.run.smithi012.stderr: 73150K .......... .......... .......... .......... .......... 78% 25.6M 2s 2023-12-03T11:39:26.244 INFO:teuthology.orchestra.run.smithi012.stderr: 73200K .......... .......... .......... .......... .......... 78% 18.0M 2s 2023-12-03T11:39:26.248 INFO:teuthology.orchestra.run.smithi012.stderr: 73250K .......... .......... .......... .......... .......... 78% 12.1M 2s 2023-12-03T11:39:26.251 INFO:teuthology.orchestra.run.smithi012.stderr: 73300K .......... .......... .......... .......... .......... 78% 20.6M 2s 2023-12-03T11:39:26.252 INFO:teuthology.orchestra.run.smithi012.stderr: 73350K .......... .......... .......... .......... .......... 79% 33.8M 2s 2023-12-03T11:39:26.254 INFO:teuthology.orchestra.run.smithi012.stderr: 73400K .......... .......... .......... .......... .......... 79% 25.5M 2s 2023-12-03T11:39:26.255 INFO:teuthology.orchestra.run.smithi012.stderr: 73450K .......... .......... .......... .......... .......... 79% 32.5M 2s 2023-12-03T11:39:26.259 INFO:teuthology.orchestra.run.smithi012.stderr: 73500K .......... .......... .......... .......... .......... 79% 15.9M 2s 2023-12-03T11:39:26.262 INFO:teuthology.orchestra.run.smithi012.stderr: 73550K .......... .......... .......... .......... .......... 79% 18.6M 2s 2023-12-03T11:39:26.265 INFO:teuthology.orchestra.run.smithi012.stderr: 73600K .......... .......... .......... .......... .......... 79% 12.8M 2s 2023-12-03T11:39:26.266 INFO:teuthology.orchestra.run.smithi012.stderr: 73650K .......... .......... .......... .......... .......... 79% 35.1M 2s 2023-12-03T11:39:26.268 INFO:teuthology.orchestra.run.smithi012.stderr: 73700K .......... .......... .......... .......... .......... 79% 26.7M 2s 2023-12-03T11:39:26.270 INFO:teuthology.orchestra.run.smithi012.stderr: 73750K .......... .......... .......... .......... .......... 79% 23.4M 2s 2023-12-03T11:39:26.272 INFO:teuthology.orchestra.run.smithi012.stderr: 73800K .......... .......... .......... .......... .......... 79% 20.7M 2s 2023-12-03T11:39:26.274 INFO:teuthology.orchestra.run.smithi012.stderr: 73850K .......... .......... .......... .......... .......... 79% 34.1M 2s 2023-12-03T11:39:26.278 INFO:teuthology.orchestra.run.smithi012.stderr: 73900K .......... .......... .......... .......... .......... 79% 12.1M 2s 2023-12-03T11:39:26.280 INFO:teuthology.orchestra.run.smithi012.stderr: 73950K .......... .......... .......... .......... .......... 79% 21.1M 2s 2023-12-03T11:39:26.283 INFO:teuthology.orchestra.run.smithi012.stderr: 74000K .......... .......... .......... .......... .......... 79% 19.5M 2s 2023-12-03T11:39:26.284 INFO:teuthology.orchestra.run.smithi012.stderr: 74050K .......... .......... .......... .......... .......... 79% 32.7M 2s 2023-12-03T11:39:26.286 INFO:teuthology.orchestra.run.smithi012.stderr: 74100K .......... .......... .......... .......... .......... 79% 26.1M 2s 2023-12-03T11:39:26.289 INFO:teuthology.orchestra.run.smithi012.stderr: 74150K .......... .......... .......... .......... .......... 79% 20.4M 2s 2023-12-03T11:39:26.293 INFO:teuthology.orchestra.run.smithi012.stderr: 74200K .......... .......... .......... .......... .......... 79% 12.2M 2s 2023-12-03T11:39:26.295 INFO:teuthology.orchestra.run.smithi012.stderr: 74250K .......... .......... .......... .......... .......... 79% 20.3M 2s 2023-12-03T11:39:26.297 INFO:teuthology.orchestra.run.smithi012.stderr: 74300K .......... .......... .......... .......... .......... 80% 26.1M 2s 2023-12-03T11:39:26.299 INFO:teuthology.orchestra.run.smithi012.stderr: 74350K .......... .......... .......... .......... .......... 80% 42.2M 2s 2023-12-03T11:39:26.300 INFO:teuthology.orchestra.run.smithi012.stderr: 74400K .......... .......... .......... .......... .......... 80% 20.5M 2s 2023-12-03T11:39:26.304 INFO:teuthology.orchestra.run.smithi012.stderr: 74450K .......... .......... .......... .......... .......... 80% 17.4M 2s 2023-12-03T11:39:26.307 INFO:teuthology.orchestra.run.smithi012.stderr: 74500K .......... .......... .......... .......... .......... 80% 12.1M 2s 2023-12-03T11:39:26.310 INFO:teuthology.orchestra.run.smithi012.stderr: 74550K .......... .......... .......... .......... .......... 80% 19.9M 2s 2023-12-03T11:39:26.311 INFO:teuthology.orchestra.run.smithi012.stderr: 74600K .......... .......... .......... .......... .......... 80% 34.3M 2s 2023-12-03T11:39:26.313 INFO:teuthology.orchestra.run.smithi012.stderr: 74650K .......... .......... .......... .......... .......... 80% 26.2M 2s 2023-12-03T11:39:26.314 INFO:teuthology.orchestra.run.smithi012.stderr: 74700K .......... .......... .......... .......... .......... 80% 36.2M 2s 2023-12-03T11:39:26.317 INFO:teuthology.orchestra.run.smithi012.stderr: 74750K .......... .......... .......... .......... .......... 80% 18.4M 2s 2023-12-03T11:39:26.320 INFO:teuthology.orchestra.run.smithi012.stderr: 74800K .......... .......... .......... .......... .......... 80% 14.7M 2s 2023-12-03T11:39:26.323 INFO:teuthology.orchestra.run.smithi012.stderr: 74850K .......... .......... .......... .......... .......... 80% 18.9M 2s 2023-12-03T11:39:26.325 INFO:teuthology.orchestra.run.smithi012.stderr: 74900K .......... .......... .......... .......... .......... 80% 22.7M 2s 2023-12-03T11:39:26.327 INFO:teuthology.orchestra.run.smithi012.stderr: 74950K .......... .......... .......... .......... .......... 80% 27.6M 2s 2023-12-03T11:39:26.329 INFO:teuthology.orchestra.run.smithi012.stderr: 75000K .......... .......... .......... .......... .......... 80% 22.1M 2s 2023-12-03T11:39:26.331 INFO:teuthology.orchestra.run.smithi012.stderr: 75050K .......... .......... .......... .......... .......... 80% 25.6M 2s 2023-12-03T11:39:26.333 INFO:teuthology.orchestra.run.smithi012.stderr: 75100K .......... .......... .......... .......... .......... 80% 20.4M 2s 2023-12-03T11:39:26.338 INFO:teuthology.orchestra.run.smithi012.stderr: 75150K .......... .......... .......... .......... .......... 80% 22.1M 2s 2023-12-03T11:39:26.340 INFO:teuthology.orchestra.run.smithi012.stderr: 75200K .......... .......... .......... .......... .......... 81% 11.7M 2s 2023-12-03T11:39:26.342 INFO:teuthology.orchestra.run.smithi012.stderr: 75250K .......... .......... .......... .......... .......... 81% 26.3M 2s 2023-12-03T11:39:26.343 INFO:teuthology.orchestra.run.smithi012.stderr: 75300K .......... .......... .......... .......... .......... 81% 25.9M 2s 2023-12-03T11:39:26.345 INFO:teuthology.orchestra.run.smithi012.stderr: 75350K .......... .......... .......... .......... .......... 81% 26.8M 2s 2023-12-03T11:39:26.347 INFO:teuthology.orchestra.run.smithi012.stderr: 75400K .......... .......... .......... .......... .......... 81% 24.9M 2s 2023-12-03T11:39:26.350 INFO:teuthology.orchestra.run.smithi012.stderr: 75450K .......... .......... .......... .......... .......... 81% 16.6M 2s 2023-12-03T11:39:26.353 INFO:teuthology.orchestra.run.smithi012.stderr: 75500K .......... .......... .......... .......... .......... 81% 16.9M 2s 2023-12-03T11:39:26.356 INFO:teuthology.orchestra.run.smithi012.stderr: 75550K .......... .......... .......... .......... .......... 81% 23.7M 2s 2023-12-03T11:39:26.358 INFO:teuthology.orchestra.run.smithi012.stderr: 75600K .......... .......... .......... .......... .......... 81% 17.9M 2s 2023-12-03T11:39:26.359 INFO:teuthology.orchestra.run.smithi012.stderr: 75650K .......... .......... .......... .......... .......... 81% 33.9M 2s 2023-12-03T11:39:26.362 INFO:teuthology.orchestra.run.smithi012.stderr: 75700K .......... .......... .......... .......... .......... 81% 18.1M 2s 2023-12-03T11:39:26.364 INFO:teuthology.orchestra.run.smithi012.stderr: 75750K .......... .......... .......... .......... .......... 81% 29.7M 2s 2023-12-03T11:39:26.368 INFO:teuthology.orchestra.run.smithi012.stderr: 75800K .......... .......... .......... .......... .......... 81% 11.7M 2s 2023-12-03T11:39:26.370 INFO:teuthology.orchestra.run.smithi012.stderr: 75850K .......... .......... .......... .......... .......... 81% 23.0M 2s 2023-12-03T11:39:26.372 INFO:teuthology.orchestra.run.smithi012.stderr: 75900K .......... .......... .......... .......... .......... 81% 26.9M 2s 2023-12-03T11:39:26.374 INFO:teuthology.orchestra.run.smithi012.stderr: 75950K .......... .......... .......... .......... .......... 81% 32.0M 2s 2023-12-03T11:39:26.376 INFO:teuthology.orchestra.run.smithi012.stderr: 76000K .......... .......... .......... .......... .......... 81% 20.7M 2s 2023-12-03T11:39:26.379 INFO:teuthology.orchestra.run.smithi012.stderr: 76050K .......... .......... .......... .......... .......... 81% 19.5M 2s 2023-12-03T11:39:26.382 INFO:teuthology.orchestra.run.smithi012.stderr: 76100K .......... .......... .......... .......... .......... 81% 14.2M 2s 2023-12-03T11:39:26.384 INFO:teuthology.orchestra.run.smithi012.stderr: 76150K .......... .......... .......... .......... .......... 82% 18.7M 2s 2023-12-03T11:39:26.385 INFO:teuthology.orchestra.run.smithi012.stderr: 76200K .......... .......... .......... .......... .......... 82% 37.1M 2s 2023-12-03T11:39:26.388 INFO:teuthology.orchestra.run.smithi012.stderr: 76250K .......... .......... .......... .......... .......... 82% 22.0M 2s 2023-12-03T11:39:26.390 INFO:teuthology.orchestra.run.smithi012.stderr: 76300K .......... .......... .......... .......... .......... 82% 24.4M 2s 2023-12-03T11:39:26.393 INFO:teuthology.orchestra.run.smithi012.stderr: 76350K .......... .......... .......... .......... .......... 82% 24.5M 2s 2023-12-03T11:39:26.395 INFO:teuthology.orchestra.run.smithi012.stderr: 76400K .......... .......... .......... .......... .......... 82% 14.8M 2s 2023-12-03T11:39:26.397 INFO:teuthology.orchestra.run.smithi012.stderr: 76450K .......... .......... .......... .......... .......... 82% 26.1M 2s 2023-12-03T11:39:26.402 INFO:teuthology.orchestra.run.smithi012.stderr: 76500K .......... .......... .......... .......... .......... 82% 15.0M 2s 2023-12-03T11:39:26.403 INFO:teuthology.orchestra.run.smithi012.stderr: 76550K .......... .......... .......... .......... .......... 82% 19.8M 2s 2023-12-03T11:39:26.404 INFO:teuthology.orchestra.run.smithi012.stderr: 76600K .......... .......... .......... .......... .......... 82% 34.4M 2s 2023-12-03T11:39:26.405 INFO:teuthology.orchestra.run.smithi012.stderr: 76650K .......... .......... .......... .......... .......... 82% 34.1M 2s 2023-12-03T11:39:26.408 INFO:teuthology.orchestra.run.smithi012.stderr: 76700K .......... .......... .......... .......... .......... 82% 19.6M 2s 2023-12-03T11:39:26.412 INFO:teuthology.orchestra.run.smithi012.stderr: 76750K .......... .......... .......... .......... .......... 82% 15.0M 2s 2023-12-03T11:39:26.414 INFO:teuthology.orchestra.run.smithi012.stderr: 76800K .......... .......... .......... .......... .......... 82% 15.4M 2s 2023-12-03T11:39:26.416 INFO:teuthology.orchestra.run.smithi012.stderr: 76850K .......... .......... .......... .......... .......... 82% 26.1M 2s 2023-12-03T11:39:26.419 INFO:teuthology.orchestra.run.smithi012.stderr: 76900K .......... .......... .......... .......... .......... 82% 19.9M 2s 2023-12-03T11:39:26.420 INFO:teuthology.orchestra.run.smithi012.stderr: 76950K .......... .......... .......... .......... .......... 82% 61.2M 2s 2023-12-03T11:39:26.423 INFO:teuthology.orchestra.run.smithi012.stderr: 77000K .......... .......... .......... .......... .......... 82% 14.9M 1s 2023-12-03T11:39:26.425 INFO:teuthology.orchestra.run.smithi012.stderr: 77050K .......... .......... .......... .......... .......... 82% 25.9M 1s 2023-12-03T11:39:26.429 INFO:teuthology.orchestra.run.smithi012.stderr: 77100K .......... .......... .......... .......... .......... 83% 13.2M 1s 2023-12-03T11:39:26.431 INFO:teuthology.orchestra.run.smithi012.stderr: 77150K .......... .......... .......... .......... .......... 83% 33.8M 1s 2023-12-03T11:39:26.433 INFO:teuthology.orchestra.run.smithi012.stderr: 77200K .......... .......... .......... .......... .......... 83% 18.5M 1s 2023-12-03T11:39:26.434 INFO:teuthology.orchestra.run.smithi012.stderr: 77250K .......... .......... .......... .......... .......... 83% 28.0M 1s 2023-12-03T11:39:26.436 INFO:teuthology.orchestra.run.smithi012.stderr: 77300K .......... .......... .......... .......... .......... 83% 25.3M 1s 2023-12-03T11:39:26.440 INFO:teuthology.orchestra.run.smithi012.stderr: 77350K .......... .......... .......... .......... .......... 83% 14.1M 1s 2023-12-03T11:39:26.441 INFO:teuthology.orchestra.run.smithi012.stderr: 77400K .......... .......... .......... .......... .......... 83% 29.4M 1s 2023-12-03T11:39:26.445 INFO:teuthology.orchestra.run.smithi012.stderr: 77450K .......... .......... .......... .......... .......... 83% 15.4M 1s 2023-12-03T11:39:26.446 INFO:teuthology.orchestra.run.smithi012.stderr: 77500K .......... .......... .......... .......... .......... 83% 27.9M 1s 2023-12-03T11:39:26.449 INFO:teuthology.orchestra.run.smithi012.stderr: 77550K .......... .......... .......... .......... .......... 83% 21.3M 1s 2023-12-03T11:39:26.451 INFO:teuthology.orchestra.run.smithi012.stderr: 77600K .......... .......... .......... .......... .......... 83% 19.8M 1s 2023-12-03T11:39:26.453 INFO:teuthology.orchestra.run.smithi012.stderr: 77650K .......... .......... .......... .......... .......... 83% 25.9M 1s 2023-12-03T11:39:26.456 INFO:teuthology.orchestra.run.smithi012.stderr: 77700K .......... .......... .......... .......... .......... 83% 15.7M 1s 2023-12-03T11:39:26.458 INFO:teuthology.orchestra.run.smithi012.stderr: 77750K .......... .......... .......... .......... .......... 83% 22.0M 1s 2023-12-03T11:39:26.461 INFO:teuthology.orchestra.run.smithi012.stderr: 77800K .......... .......... .......... .......... .......... 83% 16.7M 1s 2023-12-03T11:39:26.463 INFO:teuthology.orchestra.run.smithi012.stderr: 77850K .......... .......... .......... .......... .......... 83% 29.4M 1s 2023-12-03T11:39:26.464 INFO:teuthology.orchestra.run.smithi012.stderr: 77900K .......... .......... .......... .......... .......... 83% 32.5M 1s 2023-12-03T11:39:26.467 INFO:teuthology.orchestra.run.smithi012.stderr: 77950K .......... .......... .......... .......... .......... 83% 28.1M 1s 2023-12-03T11:39:26.470 INFO:teuthology.orchestra.run.smithi012.stderr: 78000K .......... .......... .......... .......... .......... 84% 13.2M 1s 2023-12-03T11:39:26.473 INFO:teuthology.orchestra.run.smithi012.stderr: 78050K .......... .......... .......... .......... .......... 84% 15.8M 1s 2023-12-03T11:39:26.475 INFO:teuthology.orchestra.run.smithi012.stderr: 78100K .......... .......... .......... .......... .......... 84% 26.1M 1s 2023-12-03T11:39:26.477 INFO:teuthology.orchestra.run.smithi012.stderr: 78150K .......... .......... .......... .......... .......... 84% 17.9M 1s 2023-12-03T11:39:26.478 INFO:teuthology.orchestra.run.smithi012.stderr: 78200K .......... .......... .......... .......... .......... 84% 50.7M 1s 2023-12-03T11:39:26.481 INFO:teuthology.orchestra.run.smithi012.stderr: 78250K .......... .......... .......... .......... .......... 84% 19.9M 1s 2023-12-03T11:39:26.482 INFO:teuthology.orchestra.run.smithi012.stderr: 78300K .......... .......... .......... .......... .......... 84% 30.9M 1s 2023-12-03T11:39:26.486 INFO:teuthology.orchestra.run.smithi012.stderr: 78350K .......... .......... .......... .......... .......... 84% 15.2M 1s 2023-12-03T11:39:26.489 INFO:teuthology.orchestra.run.smithi012.stderr: 78400K .......... .......... .......... .......... .......... 84% 14.5M 1s 2023-12-03T11:39:26.493 INFO:teuthology.orchestra.run.smithi012.stderr: 78450K .......... .......... .......... .......... .......... 84% 15.5M 1s 2023-12-03T11:39:26.494 INFO:teuthology.orchestra.run.smithi012.stderr: 78500K .......... .......... .......... .......... .......... 84% 42.9M 1s 2023-12-03T11:39:26.495 INFO:teuthology.orchestra.run.smithi012.stderr: 78550K .......... .......... .......... .......... .......... 84% 26.7M 1s 2023-12-03T11:39:26.497 INFO:teuthology.orchestra.run.smithi012.stderr: 78600K .......... .......... .......... .......... .......... 84% 30.5M 1s 2023-12-03T11:39:26.501 INFO:teuthology.orchestra.run.smithi012.stderr: 78650K .......... .......... .......... .......... .......... 84% 13.3M 1s 2023-12-03T11:39:26.503 INFO:teuthology.orchestra.run.smithi012.stderr: 78700K .......... .......... .......... .......... .......... 84% 20.9M 1s 2023-12-03T11:39:26.506 INFO:teuthology.orchestra.run.smithi012.stderr: 78750K .......... .......... .......... .......... .......... 84% 28.1M 1s 2023-12-03T11:39:26.508 INFO:teuthology.orchestra.run.smithi012.stderr: 78800K .......... .......... .......... .......... .......... 84% 15.6M 1s 2023-12-03T11:39:26.509 INFO:teuthology.orchestra.run.smithi012.stderr: 78850K .......... .......... .......... .......... .......... 84% 30.5M 1s 2023-12-03T11:39:26.512 INFO:teuthology.orchestra.run.smithi012.stderr: 78900K .......... .......... .......... .......... .......... 84% 22.5M 1s 2023-12-03T11:39:26.514 INFO:teuthology.orchestra.run.smithi012.stderr: 78950K .......... .......... .......... .......... .......... 85% 19.9M 1s 2023-12-03T11:39:26.516 INFO:teuthology.orchestra.run.smithi012.stderr: 79000K .......... .......... .......... .......... .......... 85% 25.6M 1s 2023-12-03T11:39:26.519 INFO:teuthology.orchestra.run.smithi012.stderr: 79050K .......... .......... .......... .......... .......... 85% 16.5M 1s 2023-12-03T11:39:26.523 INFO:teuthology.orchestra.run.smithi012.stderr: 79100K .......... .......... .......... .......... .......... 85% 14.3M 1s 2023-12-03T11:39:26.524 INFO:teuthology.orchestra.run.smithi012.stderr: 79150K .......... .......... .......... .......... .......... 85% 42.9M 1s 2023-12-03T11:39:26.526 INFO:teuthology.orchestra.run.smithi012.stderr: 79200K .......... .......... .......... .......... .......... 85% 20.8M 1s 2023-12-03T11:39:26.528 INFO:teuthology.orchestra.run.smithi012.stderr: 79250K .......... .......... .......... .......... .......... 85% 22.2M 1s 2023-12-03T11:39:26.531 INFO:teuthology.orchestra.run.smithi012.stderr: 79300K .......... .......... .......... .......... .......... 85% 17.0M 1s 2023-12-03T11:39:26.534 INFO:teuthology.orchestra.run.smithi012.stderr: 79350K .......... .......... .......... .......... .......... 85% 15.4M 1s 2023-12-03T11:39:26.536 INFO:teuthology.orchestra.run.smithi012.stderr: 79400K .......... .......... .......... .......... .......... 85% 23.5M 1s 2023-12-03T11:39:26.538 INFO:teuthology.orchestra.run.smithi012.stderr: 79450K .......... .......... .......... .......... .......... 85% 24.6M 1s 2023-12-03T11:39:26.539 INFO:teuthology.orchestra.run.smithi012.stderr: 79500K .......... .......... .......... .......... .......... 85% 41.2M 1s 2023-12-03T11:39:26.542 INFO:teuthology.orchestra.run.smithi012.stderr: 79550K .......... .......... .......... .......... .......... 85% 17.7M 1s 2023-12-03T11:39:26.546 INFO:teuthology.orchestra.run.smithi012.stderr: 79600K .......... .......... .......... .......... .......... 85% 13.4M 1s 2023-12-03T11:39:26.548 INFO:teuthology.orchestra.run.smithi012.stderr: 79650K .......... .......... .......... .......... .......... 85% 18.0M 1s 2023-12-03T11:39:26.551 INFO:teuthology.orchestra.run.smithi012.stderr: 79700K .......... .......... .......... .......... .......... 85% 16.2M 1s 2023-12-03T11:39:26.552 INFO:teuthology.orchestra.run.smithi012.stderr: 79750K .......... .......... .......... .......... .......... 85% 47.7M 1s 2023-12-03T11:39:26.554 INFO:teuthology.orchestra.run.smithi012.stderr: 79800K .......... .......... .......... .......... .......... 85% 32.0M 1s 2023-12-03T11:39:26.556 INFO:teuthology.orchestra.run.smithi012.stderr: 79850K .......... .......... .......... .......... .......... 86% 29.6M 1s 2023-12-03T11:39:26.559 INFO:teuthology.orchestra.run.smithi012.stderr: 79900K .......... .......... .......... .......... .......... 86% 14.8M 1s 2023-12-03T11:39:26.563 INFO:teuthology.orchestra.run.smithi012.stderr: 79950K .......... .......... .......... .......... .......... 86% 30.9M 1s 2023-12-03T11:39:26.564 INFO:teuthology.orchestra.run.smithi012.stderr: 80000K .......... .......... .......... .......... .......... 86% 12.2M 1s 2023-12-03T11:39:26.567 INFO:teuthology.orchestra.run.smithi012.stderr: 80050K .......... .......... .......... .......... .......... 86% 17.8M 1s 2023-12-03T11:39:26.568 INFO:teuthology.orchestra.run.smithi012.stderr: 80100K .......... .......... .......... .......... .......... 86% 45.6M 1s 2023-12-03T11:39:26.570 INFO:teuthology.orchestra.run.smithi012.stderr: 80150K .......... .......... .......... .......... .......... 86% 22.0M 1s 2023-12-03T11:39:26.572 INFO:teuthology.orchestra.run.smithi012.stderr: 80200K .......... .......... .......... .......... .......... 86% 24.7M 1s 2023-12-03T11:39:26.575 INFO:teuthology.orchestra.run.smithi012.stderr: 80250K .......... .......... .......... .......... .......... 86% 18.2M 1s 2023-12-03T11:39:26.578 INFO:teuthology.orchestra.run.smithi012.stderr: 80300K .......... .......... .......... .......... .......... 86% 16.7M 1s 2023-12-03T11:39:26.581 INFO:teuthology.orchestra.run.smithi012.stderr: 80350K .......... .......... .......... .......... .......... 86% 16.8M 1s 2023-12-03T11:39:26.583 INFO:teuthology.orchestra.run.smithi012.stderr: 80400K .......... .......... .......... .......... .......... 86% 25.7M 1s 2023-12-03T11:39:26.585 INFO:teuthology.orchestra.run.smithi012.stderr: 80450K .......... .......... .......... .......... .......... 86% 26.5M 1s 2023-12-03T11:39:26.586 INFO:teuthology.orchestra.run.smithi012.stderr: 80500K .......... .......... .......... .......... .......... 86% 24.8M 1s 2023-12-03T11:39:26.589 INFO:teuthology.orchestra.run.smithi012.stderr: 80550K .......... .......... .......... .......... .......... 86% 23.3M 1s 2023-12-03T11:39:26.593 INFO:teuthology.orchestra.run.smithi012.stderr: 80600K .......... .......... .......... .......... .......... 86% 11.5M 1s 2023-12-03T11:39:26.594 INFO:teuthology.orchestra.run.smithi012.stderr: 80650K .......... .......... .......... .......... .......... 86% 38.8M 1s 2023-12-03T11:39:26.596 INFO:teuthology.orchestra.run.smithi012.stderr: 80700K .......... .......... .......... .......... .......... 86% 20.1M 1s 2023-12-03T11:39:26.599 INFO:teuthology.orchestra.run.smithi012.stderr: 80750K .......... .......... .......... .......... .......... 86% 39.6M 1s 2023-12-03T11:39:26.600 INFO:teuthology.orchestra.run.smithi012.stderr: 80800K .......... .......... .......... .......... .......... 87% 20.7M 1s 2023-12-03T11:39:26.603 INFO:teuthology.orchestra.run.smithi012.stderr: 80850K .......... .......... .......... .......... .......... 87% 15.7M 1s 2023-12-03T11:39:26.605 INFO:teuthology.orchestra.run.smithi012.stderr: 80900K .......... .......... .......... .......... .......... 87% 30.7M 1s 2023-12-03T11:39:26.608 INFO:teuthology.orchestra.run.smithi012.stderr: 80950K .......... .......... .......... .......... .......... 87% 13.7M 1s 2023-12-03T11:39:26.610 INFO:teuthology.orchestra.run.smithi012.stderr: 81000K .......... .......... .......... .......... .......... 87% 25.8M 1s 2023-12-03T11:39:26.612 INFO:teuthology.orchestra.run.smithi012.stderr: 81050K .......... .......... .......... .......... .......... 87% 24.7M 1s 2023-12-03T11:39:26.614 INFO:teuthology.orchestra.run.smithi012.stderr: 81100K .......... .......... .......... .......... .......... 87% 29.0M 1s 2023-12-03T11:39:26.616 INFO:teuthology.orchestra.run.smithi012.stderr: 81150K .......... .......... .......... .......... .......... 87% 22.9M 1s 2023-12-03T11:39:26.620 INFO:teuthology.orchestra.run.smithi012.stderr: 81200K .......... .......... .......... .......... .......... 87% 15.0M 1s 2023-12-03T11:39:26.623 INFO:teuthology.orchestra.run.smithi012.stderr: 81250K .......... .......... .......... .......... .......... 87% 12.9M 1s 2023-12-03T11:39:26.625 INFO:teuthology.orchestra.run.smithi012.stderr: 81300K .......... .......... .......... .......... .......... 87% 25.3M 1s 2023-12-03T11:39:26.626 INFO:teuthology.orchestra.run.smithi012.stderr: 81350K .......... .......... .......... .......... .......... 87% 35.2M 1s 2023-12-03T11:39:26.629 INFO:teuthology.orchestra.run.smithi012.stderr: 81400K .......... .......... .......... .......... .......... 87% 24.0M 1s 2023-12-03T11:39:26.631 INFO:teuthology.orchestra.run.smithi012.stderr: 81450K .......... .......... .......... .......... .......... 87% 20.4M 1s 2023-12-03T11:39:26.637 INFO:teuthology.orchestra.run.smithi012.stderr: 81500K .......... .......... .......... .......... .......... 87% 9.29M 1s 2023-12-03T11:39:26.637 INFO:teuthology.orchestra.run.smithi012.stderr: 81550K .......... .......... .......... .......... .......... 87% 344M 1s 2023-12-03T11:39:26.638 INFO:teuthology.orchestra.run.smithi012.stderr: 81600K .......... .......... .......... .......... .......... 87% 22.1M 1s 2023-12-03T11:39:26.641 INFO:teuthology.orchestra.run.smithi012.stderr: 81650K .......... .......... .......... .......... .......... 87% 22.7M 1s 2023-12-03T11:39:26.644 INFO:teuthology.orchestra.run.smithi012.stderr: 81700K .......... .......... .......... .......... .......... 87% 17.9M 1s 2023-12-03T11:39:26.644 INFO:teuthology.orchestra.run.smithi012.stderr: 81750K .......... .......... .......... .......... .......... 88% 54.2M 1s 2023-12-03T11:39:26.646 INFO:teuthology.orchestra.run.smithi012.stderr: 81800K .......... .......... .......... .......... .......... 88% 26.3M 1s 2023-12-03T11:39:26.652 INFO:teuthology.orchestra.run.smithi012.stderr: 81850K .......... .......... .......... .......... .......... 88% 9.07M 1s 2023-12-03T11:39:26.653 INFO:teuthology.orchestra.run.smithi012.stderr: 81900K .......... .......... .......... .......... .......... 88% 25.1M 1s 2023-12-03T11:39:26.656 INFO:teuthology.orchestra.run.smithi012.stderr: 81950K .......... .......... .......... .......... .......... 88% 29.7M 1s 2023-12-03T11:39:26.658 INFO:teuthology.orchestra.run.smithi012.stderr: 82000K .......... .......... .......... .......... .......... 88% 20.7M 1s 2023-12-03T11:39:26.659 INFO:teuthology.orchestra.run.smithi012.stderr: 82050K .......... .......... .......... .......... .......... 88% 26.5M 1s 2023-12-03T11:39:26.661 INFO:teuthology.orchestra.run.smithi012.stderr: 82100K .......... .......... .......... .......... .......... 88% 33.6M 1s 2023-12-03T11:39:26.665 INFO:teuthology.orchestra.run.smithi012.stderr: 82150K .......... .......... .......... .......... .......... 88% 11.9M 1s 2023-12-03T11:39:26.668 INFO:teuthology.orchestra.run.smithi012.stderr: 82200K .......... .......... .......... .......... .......... 88% 25.4M 1s 2023-12-03T11:39:26.669 INFO:teuthology.orchestra.run.smithi012.stderr: 82250K .......... .......... .......... .......... .......... 88% 20.0M 1s 2023-12-03T11:39:26.670 INFO:teuthology.orchestra.run.smithi012.stderr: 82300K .......... .......... .......... .......... .......... 88% 48.4M 1s 2023-12-03T11:39:26.672 INFO:teuthology.orchestra.run.smithi012.stderr: 82350K .......... .......... .......... .......... .......... 88% 23.4M 1s 2023-12-03T11:39:26.676 INFO:teuthology.orchestra.run.smithi012.stderr: 82400K .......... .......... .......... .......... .......... 88% 16.5M 1s 2023-12-03T11:39:26.677 INFO:teuthology.orchestra.run.smithi012.stderr: 82450K .......... .......... .......... .......... .......... 88% 31.5M 1s 2023-12-03T11:39:26.682 INFO:teuthology.orchestra.run.smithi012.stderr: 82500K .......... .......... .......... .......... .......... 88% 11.5M 1s 2023-12-03T11:39:26.682 INFO:teuthology.orchestra.run.smithi012.stderr: 82550K .......... .......... .......... .......... .......... 88% 76.2M 1s 2023-12-03T11:39:26.685 INFO:teuthology.orchestra.run.smithi012.stderr: 82600K .......... .......... .......... .......... .......... 88% 15.7M 1s 2023-12-03T11:39:26.686 INFO:teuthology.orchestra.run.smithi012.stderr: 82650K .......... .......... .......... .......... .......... 89% 46.9M 1s 2023-12-03T11:39:26.688 INFO:teuthology.orchestra.run.smithi012.stderr: 82700K .......... .......... .......... .......... .......... 89% 20.7M 1s 2023-12-03T11:39:26.691 INFO:teuthology.orchestra.run.smithi012.stderr: 82750K .......... .......... .......... .......... .......... 89% 33.8M 1s 2023-12-03T11:39:26.692 INFO:teuthology.orchestra.run.smithi012.stderr: 82800K .......... .......... .......... .......... .......... 89% 23.1M 1s 2023-12-03T11:39:26.696 INFO:teuthology.orchestra.run.smithi012.stderr: 82850K .......... .......... .......... .......... .......... 89% 10.7M 1s 2023-12-03T11:39:26.699 INFO:teuthology.orchestra.run.smithi012.stderr: 82900K .......... .......... .......... .......... .......... 89% 32.3M 1s 2023-12-03T11:39:26.700 INFO:teuthology.orchestra.run.smithi012.stderr: 82950K .......... .......... .......... .......... .......... 89% 22.5M 1s 2023-12-03T11:39:26.701 INFO:teuthology.orchestra.run.smithi012.stderr: 83000K .......... .......... .......... .......... .......... 89% 45.7M 1s 2023-12-03T11:39:26.705 INFO:teuthology.orchestra.run.smithi012.stderr: 83050K .......... .......... .......... .......... .......... 89% 14.4M 1s 2023-12-03T11:39:26.705 INFO:teuthology.orchestra.run.smithi012.stderr: 83100K .......... .......... .......... .......... .......... 89% 52.3M 1s 2023-12-03T11:39:26.710 INFO:teuthology.orchestra.run.smithi012.stderr: 83150K .......... .......... .......... .......... .......... 89% 12.7M 1s 2023-12-03T11:39:26.712 INFO:teuthology.orchestra.run.smithi012.stderr: 83200K .......... .......... .......... .......... .......... 89% 17.8M 1s 2023-12-03T11:39:26.714 INFO:teuthology.orchestra.run.smithi012.stderr: 83250K .......... .......... .......... .......... .......... 89% 50.6M 1s 2023-12-03T11:39:26.715 INFO:teuthology.orchestra.run.smithi012.stderr: 83300K .......... .......... .......... .......... .......... 89% 22.5M 1s 2023-12-03T11:39:26.717 INFO:teuthology.orchestra.run.smithi012.stderr: 83350K .......... .......... .......... .......... .......... 89% 24.1M 1s 2023-12-03T11:39:26.720 INFO:teuthology.orchestra.run.smithi012.stderr: 83400K .......... .......... .......... .......... .......... 89% 18.3M 1s 2023-12-03T11:39:26.722 INFO:teuthology.orchestra.run.smithi012.stderr: 83450K .......... .......... .......... .......... .......... 89% 29.8M 1s 2023-12-03T11:39:26.727 INFO:teuthology.orchestra.run.smithi012.stderr: 83500K .......... .......... .......... .......... .......... 89% 11.3M 1s 2023-12-03T11:39:26.727 INFO:teuthology.orchestra.run.smithi012.stderr: 83550K .......... .......... .......... .......... .......... 89% 72.8M 1s 2023-12-03T11:39:26.729 INFO:teuthology.orchestra.run.smithi012.stderr: 83600K .......... .......... .......... .......... .......... 90% 21.2M 1s 2023-12-03T11:39:26.732 INFO:teuthology.orchestra.run.smithi012.stderr: 83650K .......... .......... .......... .......... .......... 90% 15.3M 1s 2023-12-03T11:39:26.733 INFO:teuthology.orchestra.run.smithi012.stderr: 83700K .......... .......... .......... .......... .......... 90% 39.2M 1s 2023-12-03T11:39:26.735 INFO:teuthology.orchestra.run.smithi012.stderr: 83750K .......... .......... .......... .......... .......... 90% 27.2M 1s 2023-12-03T11:39:26.737 INFO:teuthology.orchestra.run.smithi012.stderr: 83800K .......... .......... .......... .......... .......... 90% 27.3M 1s 2023-12-03T11:39:26.740 INFO:teuthology.orchestra.run.smithi012.stderr: 83850K .......... .......... .......... .......... .......... 90% 13.0M 1s 2023-12-03T11:39:26.742 INFO:teuthology.orchestra.run.smithi012.stderr: 83900K .......... .......... .......... .......... .......... 90% 27.7M 1s 2023-12-03T11:39:26.744 INFO:teuthology.orchestra.run.smithi012.stderr: 83950K .......... .......... .......... .......... .......... 90% 33.3M 1s 2023-12-03T11:39:26.747 INFO:teuthology.orchestra.run.smithi012.stderr: 84000K .......... .......... .......... .......... .......... 90% 15.8M 1s 2023-12-03T11:39:26.749 INFO:teuthology.orchestra.run.smithi012.stderr: 84050K .......... .......... .......... .......... .......... 90% 34.2M 1s 2023-12-03T11:39:26.751 INFO:teuthology.orchestra.run.smithi012.stderr: 84100K .......... .......... .......... .......... .......... 90% 20.0M 1s 2023-12-03T11:39:26.752 INFO:teuthology.orchestra.run.smithi012.stderr: 84150K .......... .......... .......... .......... .......... 90% 36.3M 1s 2023-12-03T11:39:26.755 INFO:teuthology.orchestra.run.smithi012.stderr: 84200K .......... .......... .......... .......... .......... 90% 14.9M 1s 2023-12-03T11:39:26.758 INFO:teuthology.orchestra.run.smithi012.stderr: 84250K .......... .......... .......... .......... .......... 90% 18.2M 1s 2023-12-03T11:39:26.760 INFO:teuthology.orchestra.run.smithi012.stderr: 84300K .......... .......... .......... .......... .......... 90% 29.5M 1s 2023-12-03T11:39:26.762 INFO:teuthology.orchestra.run.smithi012.stderr: 84350K .......... .......... .......... .......... .......... 90% 21.8M 1s 2023-12-03T11:39:26.764 INFO:teuthology.orchestra.run.smithi012.stderr: 84400K .......... .......... .......... .......... .......... 90% 24.7M 1s 2023-12-03T11:39:26.766 INFO:teuthology.orchestra.run.smithi012.stderr: 84450K .......... .......... .......... .......... .......... 90% 24.6M 1s 2023-12-03T11:39:26.769 INFO:teuthology.orchestra.run.smithi012.stderr: 84500K .......... .......... .......... .......... .......... 91% 17.5M 1s 2023-12-03T11:39:26.771 INFO:teuthology.orchestra.run.smithi012.stderr: 84550K .......... .......... .......... .......... .......... 91% 21.7M 1s 2023-12-03T11:39:26.773 INFO:teuthology.orchestra.run.smithi012.stderr: 84600K .......... .......... .......... .......... .......... 91% 23.3M 1s 2023-12-03T11:39:26.775 INFO:teuthology.orchestra.run.smithi012.stderr: 84650K .......... .......... .......... .......... .......... 91% 27.1M 1s 2023-12-03T11:39:26.777 INFO:teuthology.orchestra.run.smithi012.stderr: 84700K .......... .......... .......... .......... .......... 91% 21.8M 1s 2023-12-03T11:39:26.780 INFO:teuthology.orchestra.run.smithi012.stderr: 84750K .......... .......... .......... .......... .......... 91% 28.5M 1s 2023-12-03T11:39:26.781 INFO:teuthology.orchestra.run.smithi012.stderr: 84800K .......... .......... .......... .......... .......... 91% 19.7M 1s 2023-12-03T11:39:26.784 INFO:teuthology.orchestra.run.smithi012.stderr: 84850K .......... .......... .......... .......... .......... 91% 20.1M 1s 2023-12-03T11:39:26.786 INFO:teuthology.orchestra.run.smithi012.stderr: 84900K .......... .......... .......... .......... .......... 91% 21.7M 1s 2023-12-03T11:39:26.789 INFO:teuthology.orchestra.run.smithi012.stderr: 84950K .......... .......... .......... .......... .......... 91% 19.3M 1s 2023-12-03T11:39:26.790 INFO:teuthology.orchestra.run.smithi012.stderr: 85000K .......... .......... .......... .......... .......... 91% 31.0M 1s 2023-12-03T11:39:26.794 INFO:teuthology.orchestra.run.smithi012.stderr: 85050K .......... .......... .......... .......... .......... 91% 16.8M 1s 2023-12-03T11:39:26.794 INFO:teuthology.orchestra.run.smithi012.stderr: 85100K .......... .......... .......... .......... .......... 91% 54.4M 1s 2023-12-03T11:39:26.796 INFO:teuthology.orchestra.run.smithi012.stderr: 85150K .......... .......... .......... .......... .......... 91% 21.1M 1s 2023-12-03T11:39:26.800 INFO:teuthology.orchestra.run.smithi012.stderr: 85200K .......... .......... .......... .......... .......... 91% 13.9M 1s 2023-12-03T11:39:26.802 INFO:teuthology.orchestra.run.smithi012.stderr: 85250K .......... .......... .......... .......... .......... 91% 21.8M 1s 2023-12-03T11:39:26.805 INFO:teuthology.orchestra.run.smithi012.stderr: 85300K .......... .......... .......... .......... .......... 91% 19.9M 1s 2023-12-03T11:39:26.805 INFO:teuthology.orchestra.run.smithi012.stderr: 85350K .......... .......... .......... .......... .......... 91% 88.4M 1s 2023-12-03T11:39:26.811 INFO:teuthology.orchestra.run.smithi012.stderr: 85400K .......... .......... .......... .......... .......... 91% 15.0M 1s 2023-12-03T11:39:26.811 INFO:teuthology.orchestra.run.smithi012.stderr: 85450K .......... .......... .......... .......... .......... 92% 31.6M 1s 2023-12-03T11:39:26.812 INFO:teuthology.orchestra.run.smithi012.stderr: 85500K .......... .......... .......... .......... .......... 92% 25.2M 1s 2023-12-03T11:39:26.816 INFO:teuthology.orchestra.run.smithi012.stderr: 85550K .......... .......... .......... .......... .......... 92% 17.6M 1s 2023-12-03T11:39:26.817 INFO:teuthology.orchestra.run.smithi012.stderr: 85600K .......... .......... .......... .......... .......... 92% 19.9M 1s 2023-12-03T11:39:26.819 INFO:teuthology.orchestra.run.smithi012.stderr: 85650K .......... .......... .......... .......... .......... 92% 20.2M 1s 2023-12-03T11:39:26.822 INFO:teuthology.orchestra.run.smithi012.stderr: 85700K .......... .......... .......... .......... .......... 92% 19.1M 1s 2023-12-03T11:39:26.824 INFO:teuthology.orchestra.run.smithi012.stderr: 85750K .......... .......... .......... .......... .......... 92% 24.5M 1s 2023-12-03T11:39:26.825 INFO:teuthology.orchestra.run.smithi012.stderr: 85800K .......... .......... .......... .......... .......... 92% 63.1M 1s 2023-12-03T11:39:26.827 INFO:teuthology.orchestra.run.smithi012.stderr: 85850K .......... .......... .......... .......... .......... 92% 24.0M 1s 2023-12-03T11:39:26.829 INFO:teuthology.orchestra.run.smithi012.stderr: 85900K .......... .......... .......... .......... .......... 92% 17.0M 1s 2023-12-03T11:39:26.833 INFO:teuthology.orchestra.run.smithi012.stderr: 85950K .......... .......... .......... .......... .......... 92% 17.6M 1s 2023-12-03T11:39:26.835 INFO:teuthology.orchestra.run.smithi012.stderr: 86000K .......... .......... .......... .......... .......... 92% 22.4M 1s 2023-12-03T11:39:26.837 INFO:teuthology.orchestra.run.smithi012.stderr: 86050K .......... .......... .......... .......... .......... 92% 19.4M 1s 2023-12-03T11:39:26.839 INFO:teuthology.orchestra.run.smithi012.stderr: 86100K .......... .......... .......... .......... .......... 92% 26.7M 1s 2023-12-03T11:39:26.840 INFO:teuthology.orchestra.run.smithi012.stderr: 86150K .......... .......... .......... .......... .......... 92% 31.0M 1s 2023-12-03T11:39:26.843 INFO:teuthology.orchestra.run.smithi012.stderr: 86200K .......... .......... .......... .......... .......... 92% 19.7M 1s 2023-12-03T11:39:26.845 INFO:teuthology.orchestra.run.smithi012.stderr: 86250K .......... .......... .......... .......... .......... 92% 18.4M 1s 2023-12-03T11:39:26.847 INFO:teuthology.orchestra.run.smithi012.stderr: 86300K .......... .......... .......... .......... .......... 92% 29.2M 1s 2023-12-03T11:39:26.850 INFO:teuthology.orchestra.run.smithi012.stderr: 86350K .......... .......... .......... .......... .......... 93% 22.8M 1s 2023-12-03T11:39:26.852 INFO:teuthology.orchestra.run.smithi012.stderr: 86400K .......... .......... .......... .......... .......... 93% 18.6M 1s 2023-12-03T11:39:26.854 INFO:teuthology.orchestra.run.smithi012.stderr: 86450K .......... .......... .......... .......... .......... 93% 22.3M 1s 2023-12-03T11:39:26.856 INFO:teuthology.orchestra.run.smithi012.stderr: 86500K .......... .......... .......... .......... .......... 93% 29.8M 1s 2023-12-03T11:39:26.858 INFO:teuthology.orchestra.run.smithi012.stderr: 86550K .......... .......... .......... .......... .......... 93% 24.9M 1s 2023-12-03T11:39:26.861 INFO:teuthology.orchestra.run.smithi012.stderr: 86600K .......... .......... .......... .......... .......... 93% 18.1M 1s 2023-12-03T11:39:26.863 INFO:teuthology.orchestra.run.smithi012.stderr: 86650K .......... .......... .......... .......... .......... 93% 22.3M 1s 2023-12-03T11:39:26.865 INFO:teuthology.orchestra.run.smithi012.stderr: 86700K .......... .......... .......... .......... .......... 93% 23.8M 1s 2023-12-03T11:39:26.867 INFO:teuthology.orchestra.run.smithi012.stderr: 86750K .......... .......... .......... .......... .......... 93% 22.6M 1s 2023-12-03T11:39:26.870 INFO:teuthology.orchestra.run.smithi012.stderr: 86800K .......... .......... .......... .......... .......... 93% 20.2M 1s 2023-12-03T11:39:26.872 INFO:teuthology.orchestra.run.smithi012.stderr: 86850K .......... .......... .......... .......... .......... 93% 23.0M 1s 2023-12-03T11:39:26.872 INFO:teuthology.orchestra.run.smithi012.stderr: 86900K .......... .......... .......... .......... .......... 93% 36.9M 1s 2023-12-03T11:39:26.875 INFO:teuthology.orchestra.run.smithi012.stderr: 86950K .......... .......... .......... .......... .......... 93% 17.9M 1s 2023-12-03T11:39:26.878 INFO:teuthology.orchestra.run.smithi012.stderr: 87000K .......... .......... .......... .......... .......... 93% 21.6M 1s 2023-12-03T11:39:26.881 INFO:teuthology.orchestra.run.smithi012.stderr: 87050K .......... .......... .......... .......... .......... 93% 14.6M 1s 2023-12-03T11:39:26.882 INFO:teuthology.orchestra.run.smithi012.stderr: 87100K .......... .......... .......... .......... .......... 93% 45.5M 1s 2023-12-03T11:39:26.884 INFO:teuthology.orchestra.run.smithi012.stderr: 87150K .......... .......... .......... .......... .......... 93% 26.1M 1s 2023-12-03T11:39:26.886 INFO:teuthology.orchestra.run.smithi012.stderr: 87200K .......... .......... .......... .......... .......... 93% 21.6M 1s 2023-12-03T11:39:26.888 INFO:teuthology.orchestra.run.smithi012.stderr: 87250K .......... .......... .......... .......... .......... 93% 32.1M 0s 2023-12-03T11:39:26.892 INFO:teuthology.orchestra.run.smithi012.stderr: 87300K .......... .......... .......... .......... .......... 94% 13.6M 0s 2023-12-03T11:39:26.892 INFO:teuthology.orchestra.run.smithi012.stderr: 87350K .......... .......... .......... .......... .......... 94% 39.6M 0s 2023-12-03T11:39:26.895 INFO:teuthology.orchestra.run.smithi012.stderr: 87400K .......... .......... .......... .......... .......... 94% 23.0M 0s 2023-12-03T11:39:26.897 INFO:teuthology.orchestra.run.smithi012.stderr: 87450K .......... .......... .......... .......... .......... 94% 20.5M 0s 2023-12-03T11:39:26.899 INFO:teuthology.orchestra.run.smithi012.stderr: 87500K .......... .......... .......... .......... .......... 94% 20.7M 0s 2023-12-03T11:39:26.902 INFO:teuthology.orchestra.run.smithi012.stderr: 87550K .......... .......... .......... .......... .......... 94% 35.1M 0s 2023-12-03T11:39:26.903 INFO:teuthology.orchestra.run.smithi012.stderr: 87600K .......... .......... .......... .......... .......... 94% 25.7M 0s 2023-12-03T11:39:26.907 INFO:teuthology.orchestra.run.smithi012.stderr: 87650K .......... .......... .......... .......... .......... 94% 13.4M 0s 2023-12-03T11:39:26.907 INFO:teuthology.orchestra.run.smithi012.stderr: 87700K .......... .......... .......... .......... .......... 94% 46.9M 0s 2023-12-03T11:39:26.912 INFO:teuthology.orchestra.run.smithi012.stderr: 87750K .......... .......... .......... .......... .......... 94% 12.5M 0s 2023-12-03T11:39:26.913 INFO:teuthology.orchestra.run.smithi012.stderr: 87800K .......... .......... .......... .......... .......... 94% 30.3M 0s 2023-12-03T11:39:26.914 INFO:teuthology.orchestra.run.smithi012.stderr: 87850K .......... .......... .......... .......... .......... 94% 32.4M 0s 2023-12-03T11:39:26.916 INFO:teuthology.orchestra.run.smithi012.stderr: 87900K .......... .......... .......... .......... .......... 94% 37.4M 0s 2023-12-03T11:39:26.918 INFO:teuthology.orchestra.run.smithi012.stderr: 87950K .......... .......... .......... .......... .......... 94% 25.4M 0s 2023-12-03T11:39:26.921 INFO:teuthology.orchestra.run.smithi012.stderr: 88000K .......... .......... .......... .......... .......... 94% 13.8M 0s 2023-12-03T11:39:26.923 INFO:teuthology.orchestra.run.smithi012.stderr: 88050K .......... .......... .......... .......... .......... 94% 38.7M 0s 2023-12-03T11:39:26.927 INFO:teuthology.orchestra.run.smithi012.stderr: 88100K .......... .......... .......... .......... .......... 94% 12.7M 0s 2023-12-03T11:39:26.928 INFO:teuthology.orchestra.run.smithi012.stderr: 88150K .......... .......... .......... .......... .......... 94% 30.8M 0s 2023-12-03T11:39:26.930 INFO:teuthology.orchestra.run.smithi012.stderr: 88200K .......... .......... .......... .......... .......... 94% 30.9M 0s 2023-12-03T11:39:26.931 INFO:teuthology.orchestra.run.smithi012.stderr: 88250K .......... .......... .......... .......... .......... 95% 35.5M 0s 2023-12-03T11:39:26.933 INFO:teuthology.orchestra.run.smithi012.stderr: 88300K .......... .......... .......... .......... .......... 95% 24.2M 0s 2023-12-03T11:39:26.937 INFO:teuthology.orchestra.run.smithi012.stderr: 88350K .......... .......... .......... .......... .......... 95% 23.6M 0s 2023-12-03T11:39:26.937 INFO:teuthology.orchestra.run.smithi012.stderr: 88400K .......... .......... .......... .......... .......... 95% 20.4M 0s 2023-12-03T11:39:26.941 INFO:teuthology.orchestra.run.smithi012.stderr: 88450K .......... .......... .......... .......... .......... 95% 12.8M 0s 2023-12-03T11:39:26.944 INFO:teuthology.orchestra.run.smithi012.stderr: 88500K .......... .......... .......... .......... .......... 95% 26.9M 0s 2023-12-03T11:39:26.945 INFO:teuthology.orchestra.run.smithi012.stderr: 88550K .......... .......... .......... .......... .......... 95% 29.8M 0s 2023-12-03T11:39:26.946 INFO:teuthology.orchestra.run.smithi012.stderr: 88600K .......... .......... .......... .......... .......... 95% 42.3M 0s 2023-12-03T11:39:26.948 INFO:teuthology.orchestra.run.smithi012.stderr: 88650K .......... .......... .......... .......... .......... 95% 23.4M 0s 2023-12-03T11:39:26.950 INFO:teuthology.orchestra.run.smithi012.stderr: 88700K .......... .......... .......... .......... .......... 95% 22.2M 0s 2023-12-03T11:39:26.953 INFO:teuthology.orchestra.run.smithi012.stderr: 88750K .......... .......... .......... .......... .......... 95% 14.6M 0s 2023-12-03T11:39:26.956 INFO:teuthology.orchestra.run.smithi012.stderr: 88800K .......... .......... .......... .......... .......... 95% 16.8M 0s 2023-12-03T11:39:26.958 INFO:teuthology.orchestra.run.smithi012.stderr: 88850K .......... .......... .......... .......... .......... 95% 26.5M 0s 2023-12-03T11:39:26.960 INFO:teuthology.orchestra.run.smithi012.stderr: 88900K .......... .......... .......... .......... .......... 95% 32.5M 0s 2023-12-03T11:39:26.961 INFO:teuthology.orchestra.run.smithi012.stderr: 88950K .......... .......... .......... .......... .......... 95% 37.3M 0s 2023-12-03T11:39:26.963 INFO:teuthology.orchestra.run.smithi012.stderr: 89000K .......... .......... .......... .......... .......... 95% 23.2M 0s 2023-12-03T11:39:26.965 INFO:teuthology.orchestra.run.smithi012.stderr: 89050K .......... .......... .......... .......... .......... 95% 21.2M 0s 2023-12-03T11:39:26.968 INFO:teuthology.orchestra.run.smithi012.stderr: 89100K .......... .......... .......... .......... .......... 95% 16.1M 0s 2023-12-03T11:39:26.971 INFO:teuthology.orchestra.run.smithi012.stderr: 89150K .......... .......... .......... .......... .......... 96% 41.0M 0s 2023-12-03T11:39:26.973 INFO:teuthology.orchestra.run.smithi012.stderr: 89200K .......... .......... .......... .......... .......... 96% 15.3M 0s 2023-12-03T11:39:26.975 INFO:teuthology.orchestra.run.smithi012.stderr: 89250K .......... .......... .......... .......... .......... 96% 28.5M 0s 2023-12-03T11:39:26.976 INFO:teuthology.orchestra.run.smithi012.stderr: 89300K .......... .......... .......... .......... .......... 96% 28.6M 0s 2023-12-03T11:39:26.978 INFO:teuthology.orchestra.run.smithi012.stderr: 89350K .......... .......... .......... .......... .......... 96% 25.4M 0s 2023-12-03T11:39:26.980 INFO:teuthology.orchestra.run.smithi012.stderr: 89400K .......... .......... .......... .......... .......... 96% 22.5M 0s 2023-12-03T11:39:26.983 INFO:teuthology.orchestra.run.smithi012.stderr: 89450K .......... .......... .......... .......... .......... 96% 15.9M 0s 2023-12-03T11:39:26.985 INFO:teuthology.orchestra.run.smithi012.stderr: 89500K .......... .......... .......... .......... .......... 96% 30.5M 0s 2023-12-03T11:39:26.988 INFO:teuthology.orchestra.run.smithi012.stderr: 89550K .......... .......... .......... .......... .......... 96% 18.1M 0s 2023-12-03T11:39:26.989 INFO:teuthology.orchestra.run.smithi012.stderr: 89600K .......... .......... .......... .......... .......... 96% 26.2M 0s 2023-12-03T11:39:26.992 INFO:teuthology.orchestra.run.smithi012.stderr: 89650K .......... .......... .......... .......... .......... 96% 23.7M 0s 2023-12-03T11:39:26.993 INFO:teuthology.orchestra.run.smithi012.stderr: 89700K .......... .......... .......... .......... .......... 96% 33.3M 0s 2023-12-03T11:39:26.995 INFO:teuthology.orchestra.run.smithi012.stderr: 89750K .......... .......... .......... .......... .......... 96% 21.2M 0s 2023-12-03T11:39:26.998 INFO:teuthology.orchestra.run.smithi012.stderr: 89800K .......... .......... .......... .......... .......... 96% 17.0M 0s 2023-12-03T11:39:27.001 INFO:teuthology.orchestra.run.smithi012.stderr: 89850K .......... .......... .......... .......... .......... 96% 18.4M 0s 2023-12-03T11:39:27.003 INFO:teuthology.orchestra.run.smithi012.stderr: 89900K .......... .......... .......... .......... .......... 96% 26.0M 0s 2023-12-03T11:39:27.004 INFO:teuthology.orchestra.run.smithi012.stderr: 89950K .......... .......... .......... .......... .......... 96% 36.4M 0s 2023-12-03T11:39:27.006 INFO:teuthology.orchestra.run.smithi012.stderr: 90000K .......... .......... .......... .......... .......... 96% 22.2M 0s 2023-12-03T11:39:27.008 INFO:teuthology.orchestra.run.smithi012.stderr: 90050K .......... .......... .......... .......... .......... 96% 27.5M 0s 2023-12-03T11:39:27.010 INFO:teuthology.orchestra.run.smithi012.stderr: 90100K .......... .......... .......... .......... .......... 97% 22.4M 0s 2023-12-03T11:39:27.014 INFO:teuthology.orchestra.run.smithi012.stderr: 90150K .......... .......... .......... .......... .......... 97% 14.9M 0s 2023-12-03T11:39:27.015 INFO:teuthology.orchestra.run.smithi012.stderr: 90200K .......... .......... .......... .......... .......... 97% 36.1M 0s 2023-12-03T11:39:27.018 INFO:teuthology.orchestra.run.smithi012.stderr: 90250K .......... .......... .......... .......... .......... 97% 18.1M 0s 2023-12-03T11:39:27.019 INFO:teuthology.orchestra.run.smithi012.stderr: 90300K .......... .......... .......... .......... .......... 97% 31.4M 0s 2023-12-03T11:39:27.022 INFO:teuthology.orchestra.run.smithi012.stderr: 90350K .......... .......... .......... .......... .......... 97% 35.0M 0s 2023-12-03T11:39:27.023 INFO:teuthology.orchestra.run.smithi012.stderr: 90400K .......... .......... .......... .......... .......... 97% 19.0M 0s 2023-12-03T11:39:27.027 INFO:teuthology.orchestra.run.smithi012.stderr: 90450K .......... .......... .......... .......... .......... 97% 13.1M 0s 2023-12-03T11:39:27.028 INFO:teuthology.orchestra.run.smithi012.stderr: 90500K .......... .......... .......... .......... .......... 97% 29.8M 0s 2023-12-03T11:39:27.031 INFO:teuthology.orchestra.run.smithi012.stderr: 90550K .......... .......... .......... .......... .......... 97% 20.2M 0s 2023-12-03T11:39:27.033 INFO:teuthology.orchestra.run.smithi012.stderr: 90600K .......... .......... .......... .......... .......... 97% 28.0M 0s 2023-12-03T11:39:27.034 INFO:teuthology.orchestra.run.smithi012.stderr: 90650K .......... .......... .......... .......... .......... 97% 30.9M 0s 2023-12-03T11:39:27.035 INFO:teuthology.orchestra.run.smithi012.stderr: 90700K .......... .......... .......... .......... .......... 97% 41.5M 0s 2023-12-03T11:39:27.038 INFO:teuthology.orchestra.run.smithi012.stderr: 90750K .......... .......... .......... .......... .......... 97% 19.4M 0s 2023-12-03T11:39:27.040 INFO:teuthology.orchestra.run.smithi012.stderr: 90800K .......... .......... .......... .......... .......... 97% 20.8M 0s 2023-12-03T11:39:27.043 INFO:teuthology.orchestra.run.smithi012.stderr: 90850K .......... .......... .......... .......... .......... 97% 15.8M 0s 2023-12-03T11:39:27.046 INFO:teuthology.orchestra.run.smithi012.stderr: 90900K .......... .......... .......... .......... .......... 97% 20.7M 0s 2023-12-03T11:39:27.052 INFO:teuthology.orchestra.run.smithi012.stderr: 90950K .......... .......... .......... .......... .......... 97% 7.49M 0s 2023-12-03T11:39:27.054 INFO:teuthology.orchestra.run.smithi012.stderr: 91000K .......... .......... .......... .......... .......... 98% 30.8M 0s 2023-12-03T11:39:27.055 INFO:teuthology.orchestra.run.smithi012.stderr: 91050K .......... .......... .......... .......... .......... 98% 31.0M 0s 2023-12-03T11:39:27.058 INFO:teuthology.orchestra.run.smithi012.stderr: 91100K .......... .......... .......... .......... .......... 98% 22.7M 0s 2023-12-03T11:39:27.060 INFO:teuthology.orchestra.run.smithi012.stderr: 91150K .......... .......... .......... .......... .......... 98% 43.8M 0s 2023-12-03T11:39:27.061 INFO:teuthology.orchestra.run.smithi012.stderr: 91200K .......... .......... .......... .......... .......... 98% 22.7M 0s 2023-12-03T11:39:27.063 INFO:teuthology.orchestra.run.smithi012.stderr: 91250K .......... .......... .......... .......... .......... 98% 24.5M 0s 2023-12-03T11:39:27.068 INFO:teuthology.orchestra.run.smithi012.stderr: 91300K .......... .......... .......... .......... .......... 98% 9.75M 0s 2023-12-03T11:39:27.070 INFO:teuthology.orchestra.run.smithi012.stderr: 91350K .......... .......... .......... .......... .......... 98% 30.7M 0s 2023-12-03T11:39:27.071 INFO:teuthology.orchestra.run.smithi012.stderr: 91400K .......... .......... .......... .......... .......... 98% 37.0M 0s 2023-12-03T11:39:27.072 INFO:teuthology.orchestra.run.smithi012.stderr: 91450K .......... .......... .......... .......... .......... 98% 26.3M 0s 2023-12-03T11:39:27.074 INFO:teuthology.orchestra.run.smithi012.stderr: 91500K .......... .......... .......... .......... .......... 98% 37.7M 0s 2023-12-03T11:39:27.076 INFO:teuthology.orchestra.run.smithi012.stderr: 91550K .......... .......... .......... .......... .......... 98% 27.9M 0s 2023-12-03T11:39:27.078 INFO:teuthology.orchestra.run.smithi012.stderr: 91600K .......... .......... .......... .......... .......... 98% 21.6M 0s 2023-12-03T11:39:27.082 INFO:teuthology.orchestra.run.smithi012.stderr: 91650K .......... .......... .......... .......... .......... 98% 12.3M 0s 2023-12-03T11:39:27.084 INFO:teuthology.orchestra.run.smithi012.stderr: 91700K .......... .......... .......... .......... .......... 98% 20.8M 0s 2023-12-03T11:39:27.085 INFO:teuthology.orchestra.run.smithi012.stderr: 91750K .......... .......... .......... .......... .......... 98% 47.5M 0s 2023-12-03T11:39:27.088 INFO:teuthology.orchestra.run.smithi012.stderr: 91800K .......... .......... .......... .......... .......... 98% 21.8M 0s 2023-12-03T11:39:27.089 INFO:teuthology.orchestra.run.smithi012.stderr: 91850K .......... .......... .......... .......... .......... 98% 30.6M 0s 2023-12-03T11:39:27.090 INFO:teuthology.orchestra.run.smithi012.stderr: 91900K .......... .......... .......... .......... .......... 98% 34.1M 0s 2023-12-03T11:39:27.093 INFO:teuthology.orchestra.run.smithi012.stderr: 91950K .......... .......... .......... .......... .......... 99% 35.5M 0s 2023-12-03T11:39:27.097 INFO:teuthology.orchestra.run.smithi012.stderr: 92000K .......... .......... .......... .......... .......... 99% 9.69M 0s 2023-12-03T11:39:27.099 INFO:teuthology.orchestra.run.smithi012.stderr: 92050K .......... .......... .......... .......... .......... 99% 20.2M 0s 2023-12-03T11:39:27.100 INFO:teuthology.orchestra.run.smithi012.stderr: 92100K .......... .......... .......... .......... .......... 99% 36.7M 0s 2023-12-03T11:39:27.102 INFO:teuthology.orchestra.run.smithi012.stderr: 92150K .......... .......... .......... .......... .......... 99% 28.9M 0s 2023-12-03T11:39:27.104 INFO:teuthology.orchestra.run.smithi012.stderr: 92200K .......... .......... .......... .......... .......... 99% 33.1M 0s 2023-12-03T11:39:27.105 INFO:teuthology.orchestra.run.smithi012.stderr: 92250K .......... .......... .......... .......... .......... 99% 31.1M 0s 2023-12-03T11:39:27.107 INFO:teuthology.orchestra.run.smithi012.stderr: 92300K .......... .......... .......... .......... .......... 99% 25.8M 0s 2023-12-03T11:39:27.111 INFO:teuthology.orchestra.run.smithi012.stderr: 92350K .......... .......... .......... .......... .......... 99% 16.9M 0s 2023-12-03T11:39:27.115 INFO:teuthology.orchestra.run.smithi012.stderr: 92400K .......... .......... .......... .......... .......... 99% 11.3M 0s 2023-12-03T11:39:27.116 INFO:teuthology.orchestra.run.smithi012.stderr: 92450K .......... .......... .......... .......... .......... 99% 29.9M 0s 2023-12-03T11:39:27.117 INFO:teuthology.orchestra.run.smithi012.stderr: 92500K .......... .......... .......... .......... .......... 99% 45.2M 0s 2023-12-03T11:39:27.119 INFO:teuthology.orchestra.run.smithi012.stderr: 92550K .......... .......... .......... .......... .......... 99% 26.5M 0s 2023-12-03T11:39:27.120 INFO:teuthology.orchestra.run.smithi012.stderr: 92600K .......... .......... .......... .......... .......... 99% 36.9M 0s 2023-12-03T11:39:27.123 INFO:teuthology.orchestra.run.smithi012.stderr: 92650K .......... .......... .......... .......... .......... 99% 21.6M 0s 2023-12-03T11:39:27.125 INFO:teuthology.orchestra.run.smithi012.stderr: 92700K .......... .......... .......... .......... .......... 99% 19.4M 0s 2023-12-03T11:39:27.129 INFO:teuthology.orchestra.run.smithi012.stderr: 92750K .......... .......... .......... .......... .......... 99% 17.8M 0s 2023-12-03T11:39:27.130 INFO:teuthology.orchestra.run.smithi012.stderr: 92800K .......... .......... .......... .......... .......... 99% 17.7M 0s 2023-12-03T11:39:27.132 INFO:teuthology.orchestra.run.smithi012.stderr: 92850K .......... .......... .......... .......... .......... 99% 35.6M 0s 2023-12-03T11:39:27.132 INFO:teuthology.orchestra.run.smithi012.stderr: 92900K 100% 134G=8.0s 2023-12-03T11:39:27.132 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:39:27.132 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03 11:39:27 (11.4 MB/s) - ‘gradle-6.0.1-bin.zip’ saved [95129672/95129672] 2023-12-03T11:39:27.133 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout:Archive: gradle-6.0.1-bin.zip 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: creating: /opt/gradle/gradle-6.0.1/ 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/getting-started.html 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/LICENSE 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/NOTICE 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: creating: /opt/gradle/gradle-6.0.1/init.d/ 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/init.d/readme.txt 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: creating: /opt/gradle/gradle-6.0.1/bin/ 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/bin/gradle.bat 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: creating: /opt/gradle/gradle-6.0.1/lib/ 2023-12-03T11:39:27.159 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-api-metadata-6.0.1.jar 2023-12-03T11:39:27.161 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-launcher-6.0.1.jar 2023-12-03T11:39:27.161 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-runtime-api-info-6.0.1.jar 2023-12-03T11:39:27.172 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-6.0.1.jar 2023-12-03T11:39:27.175 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-tooling-api-6.0.1.jar 2023-12-03T11:39:27.175 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-wrapper-6.0.1.jar 2023-12-03T11:39:27.175 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-installation-beacon-6.0.1.jar 2023-12-03T11:39:27.175 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-bootstrap-6.0.1.jar 2023-12-03T11:39:27.190 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-6.0.1.jar 2023-12-03T11:39:27.191 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-file-collections-6.0.1.jar 2023-12-03T11:39:27.192 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-groovy-6.0.1.jar 2023-12-03T11:39:27.193 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-execution-6.0.1.jar 2023-12-03T11:39:27.199 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-model-core-6.0.1.jar 2023-12-03T11:39:27.200 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-packaging-6.0.1.jar 2023-12-03T11:39:27.200 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-cache-6.0.1.jar 2023-12-03T11:39:27.202 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-core-api-6.0.1.jar 2023-12-03T11:39:27.204 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-persistent-cache-6.0.1.jar 2023-12-03T11:39:27.204 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-worker-processes-6.0.1.jar 2023-12-03T11:39:27.206 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-logging-6.0.1.jar 2023-12-03T11:39:27.206 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-jvm-services-6.0.1.jar 2023-12-03T11:39:27.206 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-process-services-6.0.1.jar 2023-12-03T11:39:27.207 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-resources-6.0.1.jar 2023-12-03T11:39:27.208 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-messaging-6.0.1.jar 2023-12-03T11:39:27.209 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-native-6.0.1.jar 2023-12-03T11:39:27.209 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-groovy-6.0.1.jar 2023-12-03T11:39:27.213 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-base-services-6.0.1.jar 2023-12-03T11:39:27.213 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-build-option-6.0.1.jar 2023-12-03T11:39:27.213 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-cli-6.0.1.jar 2023-12-03T11:39:27.214 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-snapshots-6.0.1.jar 2023-12-03T11:39:27.214 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-files-6.0.1.jar 2023-12-03T11:39:27.214 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-kotlin-dsl-tooling-models-6.0.1.jar 2023-12-03T11:39:27.452 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-compiler-embeddable-1.3.50-patched-for-gradle-6.0.1.jar 2023-12-03T11:39:27.452 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-hashing-6.0.1.jar 2023-12-03T11:39:27.452 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-pineapple-6.0.1.jar 2023-12-03T11:39:27.452 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/gradle-docs-6.0.1.jar 2023-12-03T11:39:27.506 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/groovy-all-1.3-2.5.8.jar 2023-12-03T11:39:27.506 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jul-to-slf4j-1.7.28.jar 2023-12-03T11:39:27.506 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/log4j-over-slf4j-1.7.28.jar 2023-12-03T11:39:27.507 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jcl-over-slf4j-1.7.28.jar 2023-12-03T11:39:27.507 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/slf4j-api-1.7.28.jar 2023-12-03T11:39:27.523 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/guava-27.1-android.jar 2023-12-03T11:39:27.524 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-io-2.6.jar 2023-12-03T11:39:27.526 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-lang-2.6.jar 2023-12-03T11:39:27.527 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-commons-7.1.jar 2023-12-03T11:39:27.527 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-analysis-7.1.jar 2023-12-03T11:39:27.527 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-tree-7.1.jar 2023-12-03T11:39:27.528 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/asm-7.1.jar 2023-12-03T11:39:27.544 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-1.10.7.jar 2023-12-03T11:39:27.544 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk8-1.3.50.jar 2023-12-03T11:39:27.544 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/javax.inject-1.jar 2023-12-03T11:39:27.544 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-common-1.3.50.jar 2023-12-03T11:39:27.545 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-1.3.50.jar 2023-12-03T11:39:27.546 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-jvm-host-embeddable-1.3.50.jar 2023-12-03T11:39:27.548 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-embeddable-1.3.50.jar 2023-12-03T11:39:27.550 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-scripting-compiler-impl-embeddable-1.3.50.jar 2023-12-03T11:39:27.550 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-sam-with-receiver-compiler-plugin-1.3.50.jar 2023-12-03T11:39:27.566 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlinx-metadata-jvm-0.1.0.jar 2023-12-03T11:39:27.570 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/commons-compress-1.19.jar 2023-12-03T11:39:27.573 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/fastutil-8.3.0-min.jar 2023-12-03T11:39:27.575 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jansi-1.18.jar 2023-12-03T11:39:27.575 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/jsr305-3.0.2.jar 2023-12-03T11:39:27.577 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kryo-2.24.0.jar 2023-12-03T11:39:27.578 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-0.18.jar 2023-12-03T11:39:27.578 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/objenesis-2.6.jar 2023-12-03T11:39:27.582 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/trove4j-1.0.20181211.jar 2023-12-03T11:39:27.584 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/xml-apis-1.4.01.jar 2023-12-03T11:39:27.602 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-reflect-1.3.50.jar 2023-12-03T11:39:27.602 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-jdk7-1.3.50.jar 2023-12-03T11:39:27.611 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-1.3.50.jar 2023-12-03T11:39:27.611 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/annotations-13.0.jar 2023-12-03T11:39:27.612 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/ant-launcher-1.10.7.jar 2023-12-03T11:39:27.612 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/minlog-1.2.jar 2023-12-03T11:39:27.612 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-script-runtime-1.3.50.jar 2023-12-03T11:39:27.619 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-daemon-embeddable-1.3.50.jar 2023-12-03T11:39:27.620 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/failureaccess-1.0.1.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/kotlin-stdlib-common-1.3.50.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libcpp-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-amd64-libstdcpp-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libcpp-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-freebsd-i386-libstdcpp-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-0.18.jar 2023-12-03T11:39:27.622 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-osx-amd64-0.18.jar 2023-12-03T11:39:27.624 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-0.18.jar 2023-12-03T11:39:27.624 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-amd64-min-0.18.jar 2023-12-03T11:39:27.624 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-0.18.jar 2023-12-03T11:39:27.624 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-windows-i386-min-0.18.jar 2023-12-03T11:39:27.624 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-aarch64-ncurses5-0.18.jar 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses5-0.18.jar 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-amd64-ncurses6-0.18.jar 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses5-0.18.jar 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/native-platform-linux-i386-ncurses6-0.18.jar 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: creating: /opt/gradle/gradle-6.0.1/lib/plugins/ 2023-12-03T11:39:27.625 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-antlr-6.0.1.jar 2023-12-03T11:39:27.626 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-code-quality-6.0.1.jar 2023-12-03T11:39:27.626 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-signing-6.0.1.jar 2023-12-03T11:39:27.626 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-native-6.0.1.jar 2023-12-03T11:39:27.627 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-native-6.0.1.jar 2023-12-03T11:39:27.628 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-native-6.0.1.jar 2023-12-03T11:39:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-native-6.0.1.jar 2023-12-03T11:39:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-provider-plugins-6.0.1.jar 2023-12-03T11:39:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-development-6.0.1.jar 2023-12-03T11:39:27.634 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-maven-6.0.1.jar 2023-12-03T11:39:27.635 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-play-6.0.1.jar 2023-12-03T11:39:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ide-6.0.1.jar 2023-12-03T11:39:27.637 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-scala-6.0.1.jar 2023-12-03T11:39:27.638 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ear-6.0.1.jar 2023-12-03T11:39:27.639 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-play-6.0.1.jar 2023-12-03T11:39:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-javascript-6.0.1.jar 2023-12-03T11:39:27.640 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-ivy-6.0.1.jar 2023-12-03T11:39:27.641 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-jacoco-6.0.1.jar 2023-12-03T11:39:27.643 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-init-6.0.1.jar 2023-12-03T11:39:27.644 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-kotlin-dsl-tooling-builders-6.0.1.jar 2023-12-03T11:39:27.647 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugins-6.0.1.jar 2023-12-03T11:39:27.647 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-test-kit-6.0.1.jar 2023-12-03T11:39:27.647 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-groovy-6.0.1.jar 2023-12-03T11:39:27.648 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-scala-6.0.1.jar 2023-12-03T11:39:27.648 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-junit-platform-6.0.1.jar 2023-12-03T11:39:27.648 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-tooling-api-builders-6.0.1.jar 2023-12-03T11:39:27.649 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-jvm-6.0.1.jar 2023-12-03T11:39:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-java-6.0.1.jar 2023-12-03T11:39:27.651 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-language-jvm-6.0.1.jar 2023-12-03T11:39:27.652 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-jvm-6.0.1.jar 2023-12-03T11:39:27.656 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-native-6.0.1.jar 2023-12-03T11:39:27.658 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-diagnostics-6.0.1.jar 2023-12-03T11:39:27.658 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-publish-6.0.1.jar 2023-12-03T11:39:27.660 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-testing-base-6.0.1.jar 2023-12-03T11:39:27.661 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-platform-base-6.0.1.jar 2023-12-03T11:39:27.661 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-plugin-use-6.0.1.jar 2023-12-03T11:39:27.668 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-instant-execution-6.0.1.jar 2023-12-03T11:39:27.668 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-version-control-6.0.1.jar 2023-12-03T11:39:27.682 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-dependency-management-6.0.1.jar 2023-12-03T11:39:27.682 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-composite-builds-6.0.1.jar 2023-12-03T11:39:27.683 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-reporting-6.0.1.jar 2023-12-03T11:39:27.684 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-s3-6.0.1.jar 2023-12-03T11:39:27.684 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-gcs-6.0.1.jar 2023-12-03T11:39:27.684 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-cache-http-6.0.1.jar 2023-12-03T11:39:27.685 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-http-6.0.1.jar 2023-12-03T11:39:27.685 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-resources-sftp-6.0.1.jar 2023-12-03T11:39:27.685 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-build-profile-6.0.1.jar 2023-12-03T11:39:27.686 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gradle-workers-6.0.1.jar 2023-12-03T11:39:27.704 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/org.eclipse.jgit-5.5.0.201909110433-r.jar 2023-12-03T11:39:27.704 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-3.0.0.jar 2023-12-03T11:39:27.704 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-http-shared-3.0.0.jar 2023-12-03T11:39:27.705 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-services-storage-v1-rev136-1.25.0.jar 2023-12-03T11:39:27.706 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-api-client-1.25.0.jar 2023-12-03T11:39:27.707 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/asm-util-7.1.jar 2023-12-03T11:39:27.715 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ivy-2.3.0.jar 2023-12-03T11:39:27.717 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-compat-3.0.4.jar 2023-12-03T11:39:27.721 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-core-3.0.4.jar 2023-12-03T11:39:27.721 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-common-0.8-20100325.jar 2023-12-03T11:39:27.721 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/pmaven-groovy-0.8-20100325.jar 2023-12-03T11:39:27.721 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-file-3.0.0.jar 2023-12-03T11:39:27.723 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-container-default-1.7.1.jar 2023-12-03T11:39:27.723 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-connector-wagon-1.13.1.jar 2023-12-03T11:39:27.725 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpg-jdk15on-1.63.jar 2023-12-03T11:39:27.733 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/rhino-1.7.10.jar 2023-12-03T11:39:27.735 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/gson-2.8.5.jar 2023-12-03T11:39:27.737 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/simple-4.1.21.jar 2023-12-03T11:39:27.737 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jatl-0.2.3.jar 2023-12-03T11:39:27.737 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-plugin-api-3.0.4.jar 2023-12-03T11:39:27.738 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-oauth-client-1.25.0.jar 2023-12-03T11:39:27.738 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-jackson2-1.25.0.jar 2023-12-03T11:39:27.740 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/google-http-client-1.25.0.jar 2023-12-03T11:39:27.744 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/testng-6.3.1.jar 2023-12-03T11:39:27.746 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/snakeyaml-1.17.jar 2023-12-03T11:39:27.747 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/dd-plist-1.21.jar 2023-12-03T11:39:27.749 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-4.12.jar 2023-12-03T11:39:27.751 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bsh-2.0b6.jar 2023-12-03T11:39:27.752 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-launcher-1.3.1.jar 2023-12-03T11:39:27.759 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-s3-1.11.633.jar 2023-12-03T11:39:27.762 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-kms-1.11.633.jar 2023-12-03T11:39:27.768 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aws-java-sdk-core-1.11.633.jar 2023-12-03T11:39:27.773 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpclient-4.5.10.jar 2023-12-03T11:39:27.776 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcifs-1.3.17.jar 2023-12-03T11:39:27.777 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/nekohtml-1.9.22.jar 2023-12-03T11:39:27.786 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xercesImpl-2.12.0.jar 2023-12-03T11:39:27.788 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jsch-0.1.55.jar 2023-12-03T11:39:27.795 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jaxb-impl-2.3.2.jar 2023-12-03T11:39:27.795 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jmespath-java-1.11.633.jar 2023-12-03T11:39:27.804 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-databind-2.9.10.jar 2023-12-03T11:39:27.806 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-core-2.9.10.jar 2023-12-03T11:39:27.806 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jackson-annotations-2.9.10.jar 2023-12-03T11:39:27.811 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/joda-time-2.10.4.jar 2023-12-03T11:39:27.816 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcpkix-jdk15on-1.61.jar 2023-12-03T11:39:27.846 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/bcprov-jdk15on-1.63.jar 2023-12-03T11:39:27.848 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/commons-codec-1.13.jar 2023-12-03T11:39:27.849 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-aether-provider-3.0.4.jar 2023-12-03T11:39:27.849 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-impl-1.13.1.jar 2023-12-03T11:39:27.851 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-util-1.13.1.jar 2023-12-03T11:39:27.851 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-spi-1.13.1.jar 2023-12-03T11:39:27.852 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/aether-api-1.13.1.jar 2023-12-03T11:39:27.852 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/hamcrest-core-1.3.jar 2023-12-03T11:39:27.854 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/httpcore-4.4.12.jar 2023-12-03T11:39:27.855 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jcommander-1.72.jar 2023-12-03T11:39:27.855 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/jzlib-1.1.3.jar 2023-12-03T11:39:27.856 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-artifact-3.0.4.jar 2023-12-03T11:39:27.856 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-builder-3.0.4.jar 2023-12-03T11:39:27.858 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-model-3.0.4.jar 2023-12-03T11:39:27.858 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-repository-metadata-3.0.4.jar 2023-12-03T11:39:27.858 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-builder-3.0.4.jar 2023-12-03T11:39:27.859 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/maven-settings-3.0.4.jar 2023-12-03T11:39:27.860 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-sec-dispatcher-1.3.jar 2023-12-03T11:39:27.860 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-cipher-1.7.jar 2023-12-03T11:39:27.860 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-classworlds-2.5.1.jar 2023-12-03T11:39:27.860 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-component-annotations-1.5.5.jar 2023-12-03T11:39:27.860 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-interpolation-1.14.jar 2023-12-03T11:39:27.861 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/wagon-provider-api-3.0.0.jar 2023-12-03T11:39:27.862 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/plexus-utils-3.1.0.jar 2023-12-03T11:39:27.863 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/xbean-reflect-3.7.jar 2023-12-03T11:39:27.867 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/ion-java-1.0.2.jar 2023-12-03T11:39:27.868 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-engine-1.3.1.jar 2023-12-03T11:39:27.869 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/junit-platform-commons-1.3.1.jar 2023-12-03T11:39:27.869 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/apiguardian-api-1.0.0.jar 2023-12-03T11:39:27.869 INFO:teuthology.orchestra.run.smithi012.stdout: inflating: /opt/gradle/gradle-6.0.1/lib/plugins/opentest4j-1.1.1.jar 2023-12-03T11:39:27.914 INFO:teuthology.orchestra.run.smithi012.stdout:export PATH=/opt/gradle/gradle-6.0.1/bin:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout:Welcome to Gradle 6.0.1! 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout:Here are the highlights of this release: 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout: - Substantial improvements in dependency management, including 2023-12-03T11:39:28.263 INFO:teuthology.orchestra.run.smithi012.stdout: - Publishing Gradle Module Metadata in addition to pom.xml 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Advanced control of transitive versions 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Support for optional features and dependencies 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Rules to tweak published metadata 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Support for Java 13 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Faster incremental Java and Groovy compilation 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - New Zinc compiler for Scala 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - VS2019 support 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: - Support for Gradle Enterprise plugin 3.0 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout:For more details see https://docs.gradle.org/6.0.1/release-notes.html 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout:------------------------------------------------------------ 2023-12-03T11:39:28.264 INFO:teuthology.orchestra.run.smithi012.stdout:Gradle 6.0.1 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:------------------------------------------------------------ 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:Build time: 2019-11-18 20:25:01 UTC 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:Revision: fad121066a68c4701acd362daf4287a7c309a0f5 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:Kotlin: 1.3.50 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:Groovy: 2.5.8 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:JVM: 1.8.0_362 (Red Hat, Inc. 25.362-b08) 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout:OS: Linux 5.14.0-388.el9.x86_64 amd64 2023-12-03T11:39:28.265 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.267 DEBUG:tasks.s3tests_java:S3 Tests Java: BEGIN 2023-12-03T11:39:28.267 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-03T11:39:28.267 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T11:39:28.267 DEBUG:tasks.s3tests_java:S3 Tests Java: Cluster config is: ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-03T11:39:28.267 DEBUG:tasks.s3tests_java:S3 Tests Java: Host is: ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T11:39:28.267 INFO:tasks.s3tests_java:S3 Tests Java: Creating S3 users... 2023-12-03T11:39:28.559 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf is {'bucket_prefix': 'test-', 's3main': {'access_key': None, 'access_secret': None, 'region': 'mexico', 'endpoint': None, 'port': None, 'display_name': None, 'email': None, 'is_secure': None, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data'}} 2023-12-03T11:39:28.559 DEBUG:tasks.s3tests_java:S3 Tests Java: Creating user tester.client.0 2023-12-03T11:39:28.559 DEBUG:tasks.s3tests_java:S3 Tests Java: s3tests_conf[s3main] is {'access_key': 'CWVBFPTZWYYRLFXQPVQW', 'access_secret': 'BiPD76VBjujZvSwreNk1QPsGrglByXJopI7jDDQGp0VdK4PrOMSgyw==', 'region': 'mexico', 'endpoint': 'smithi012.front.sepia.ceph.com:80', 'port': 80, 'display_name': 'Ms. tester.client.0', 'email': 'tester.client.0_test@test.test', 'is_secure': False, 'SSE': 'AES256', 's3dir': 'folder', 'dir': '../data', 'user_id': 'tester.client.0', 'host': 'smithi012.front.sepia.ceph.com'} 2023-12-03T11:39:28.559 DEBUG:tasks.s3tests_java:S3 Tests Java: Setion, User = s3main, tester.client.0 2023-12-03T11:39:28.559 INFO:tasks.s3tests_java:['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '-n', 'client.0', 'user', 'create', '--uid', 'tester.client.0', '--display-name', 'Ms. tester.client.0', '--access-key', 'CWVBFPTZWYYRLFXQPVQW', '--secret', 'BiPD76VBjujZvSwreNk1QPsGrglByXJopI7jDDQGp0VdK4PrOMSgyw==', '--email', 'tester.client.0_test@test.test', '--cluster', 'ceph'] 2023-12-03T11:39:28.560 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid tester.client.0 --display-name 'Ms. tester.client.0' --access-key CWVBFPTZWYYRLFXQPVQW --secret BiPD76VBjujZvSwreNk1QPsGrglByXJopI7jDDQGp0VdK4PrOMSgyw== --email tester.client.0_test@test.test --cluster ceph 2023-12-03T11:39:28.622 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:39:28.623 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:39:28.634 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.633+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 realm 2023-12-03T11:39:28.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.636 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:39:28.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.635+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.636+0000 7ff5bfc9d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:28.637 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.636+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.637+0000 7ff5bfc9d800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:39:28.638 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.637+0000 7ff5bfc9d800 20 searching for the correct realm 2023-12-03T11:39:28.650 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bdf9e640 0 -- 172.21.15.12:0/2627699468 >> [v2:172.21.15.191:6816/2577115973,v1:172.21.15.191:6817/2577115973] conn(0x7ff5b408b950 msgr2=0x7ff5b406b400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.651 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.649+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:28.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:39:28.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 zone default found 2023-12-03T11:39:28.652 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 4 Realm: () 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 zonegroup default 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.650+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.651+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.651+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.651+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.651+0000 7ff5bfc9d800 20 started sync module instance, tier type = 2023-12-03T11:39:28.653 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.651+0000 7ff5bfc9d800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=0 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=1 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=2 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=3 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=4 2023-12-03T11:39:28.662 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=5 2023-12-03T11:39:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bdf9e640 0 -- 172.21.15.12:0/2627699468 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55d0fc094720 msgr2=0x55d0fc096be0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.661+0000 7ff5bfc9d800 20 add_watcher() i=6 2023-12-03T11:39:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.662+0000 7ff5bfc9d800 20 add_watcher() i=7 2023-12-03T11:39:28.663 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.662+0000 7ff5bfc9d800 2 all 8 watchers are set, enabling cache 2023-12-03T11:39:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.664+0000 7ff582fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.664+0000 7ff5bfc9d800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:39:28.665 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.664+0000 7ff5bfc9d800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:39:28.670 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.669+0000 7ff5bfc9d800 5 note: GC not initialized 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.669+0000 7ff4f1ffb640 20 reqs_thread_entry: start 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.669+0000 7ff5bfc9d800 20 init_complete bucket index max shards: 1 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 20 Filter name: none 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff4e37fe640 20 reqs_thread_entry: start 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 10 cache get: name=default.rgw.meta+users.uid+tester.client.0 : miss 2023-12-03T11:39:28.671 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.672 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 10 adding default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 10 cache get: name=default.rgw.meta+users.email+tester.client.0_test@test.test : miss 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.670+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 adding default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 cache get: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW : miss 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW info.flags=0x0 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 adding default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW to cache LRU end 2023-12-03T11:39:28.673 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 cache get: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW : hit (negative entry) 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.671+0000 7ff5bfc9d800 10 cache get: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW : hit (negative entry) 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.672+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.672+0000 7ff5bfc9d800 10 moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.672+0000 7ff5bfc9d800 10 distributing notification oid=default.rgw.control:notify.6 cni=[op: 0, obj: default.rgw.meta:users.uid:tester.client.0, ofs0, ns] 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.673+0000 7ff59f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725955 cookie 94356070060512 notifier 7710 bl.length()=636 2023-12-03T11:39:28.674 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.673+0000 7ff59f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.uid+tester.client.0 info.flags=0x17 2023-12-03T11:39:28.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.673+0000 7ff59f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.uid+tester.client.0 to cache LRU end 2023-12-03T11:39:28.675 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.674+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2023-12-03T11:39:28.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.674+0000 7ff5bfc9d800 10 moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:39:28.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.674+0000 7ff5bfc9d800 10 distributing notification oid=default.rgw.control:notify.3 cni=[op: 0, obj: default.rgw.meta:users.email:tester.client.0_test@test.test, ofs0, ns] 2023-12-03T11:39:28.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.675+0000 7ff59f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725931 cookie 94356070053552 notifier 7710 bl.length()=200 2023-12-03T11:39:28.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.675+0000 7ff59f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.email+tester.client.0_test@test.test info.flags=0x7 2023-12-03T11:39:28.676 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.675+0000 7ff59f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.email+tester.client.0_test@test.test to cache LRU end 2023-12-03T11:39:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.676+0000 7ff5bfc9d800 10 cache put: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW info.flags=0x7 2023-12-03T11:39:28.677 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.676+0000 7ff5bfc9d800 10 moving default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW to cache LRU end 2023-12-03T11:39:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.676+0000 7ff5bfc9d800 10 distributing notification oid=default.rgw.control:notify.4 cni=[op: 0, obj: default.rgw.meta:users.keys:CWVBFPTZWYYRLFXQPVQW, ofs0, ns] 2023-12-03T11:39:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.677+0000 7ff59f7fe640 10 rgw watcher librados: RGWWatcher::handle_notify() notify_id 1086626725940 cookie 94356070058656 notifier 7710 bl.length()=189 2023-12-03T11:39:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.677+0000 7ff59f7fe640 10 rgw watcher librados: cache put: name=default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW info.flags=0x7 2023-12-03T11:39:28.678 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.677+0000 7ff59f7fe640 10 rgw watcher librados: moving default.rgw.meta+users.keys+CWVBFPTZWYYRLFXQPVQW to cache LRU end 2023-12-03T11:39:28.678 INFO:teuthology.orchestra.run.smithi012.stdout:{ 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "user_id": "tester.client.0", 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "display_name": "Ms. tester.client.0", 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "email": "tester.client.0_test@test.test", 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "suspended": 0, 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "max_buckets": 1000, 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "subusers": [], 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "keys": [ 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: { 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "user": "tester.client.0", 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "access_key": "CWVBFPTZWYYRLFXQPVQW", 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: "secret_key": "BiPD76VBjujZvSwreNk1QPsGrglByXJopI7jDDQGp0VdK4PrOMSgyw==" 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: } 2023-12-03T11:39:28.679 INFO:teuthology.orchestra.run.smithi012.stdout: ], 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "swift_keys": [], 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "caps": [], 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "op_mask": "read, write, delete", 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_placement": "", 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "default_storage_class": "", 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "placement_tags": [], 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "bucket_quota": { 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "user_quota": { 2023-12-03T11:39:28.680 INFO:teuthology.orchestra.run.smithi012.stdout: "enabled": false, 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "check_on_raw": false, 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size": -1, 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max_size_kb": 0, 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "max_objects": -1 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: }, 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "temp_url_keys": [], 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "type": "rgw", 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: "mfa_ids": [] 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout:} 2023-12-03T11:39:28.681 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:28.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.678+0000 7ff582fe5640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:28.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.680+0000 7ff5bfc9d800 20 remove_watcher() i=0 2023-12-03T11:39:28.682 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.680+0000 7ff5bfc9d800 2 removed watcher, disabling cache 2023-12-03T11:39:28.683 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.682+0000 7ff5bfc9d800 20 remove_watcher() i=1 2023-12-03T11:39:28.684 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.683+0000 7ff5bfc9d800 20 remove_watcher() i=2 2023-12-03T11:39:28.685 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.684+0000 7ff5bfc9d800 20 remove_watcher() i=3 2023-12-03T11:39:28.686 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.685+0000 7ff5bfc9d800 20 remove_watcher() i=4 2023-12-03T11:39:28.687 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.686+0000 7ff5bfc9d800 20 remove_watcher() i=5 2023-12-03T11:39:28.688 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.687+0000 7ff5bfc9d800 20 remove_watcher() i=6 2023-12-03T11:39:28.689 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.688+0000 7ff5bfc9d800 20 remove_watcher() i=7 2023-12-03T11:39:28.695 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2023-12-03T11:39:28.695 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf 2023-12-03T11:39:28.725 INFO:tasks.s3tests_java:S3 Tests Java: Running tests... 2023-12-03T11:39:28.725 DEBUG:teuthology.orchestra.run.smithi012:> cp /home/ubuntu/cephtest/archive/s3-tests-java.client.0.conf /home/ubuntu/cephtest/s3-tests-java/config.properties 2023-12-03T11:39:28.792 DEBUG:teuthology.orchestra.run.smithi012:> radosgw-admin gc process --include-all 2023-12-03T11:39:28.883 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:39:28.884 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:39:28.898 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.896+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.898+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.898+0000 7f158053b800 20 realm 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.898+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.899+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.899+0000 7f158053b800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.899+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.899+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.900 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.899+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.900+0000 7f158053b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:28.901 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.900+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.901+0000 7f158053b800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:39:28.902 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.901+0000 7f158053b800 20 searching for the correct realm 2023-12-03T11:39:28.912 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.911+0000 7f157c240640 0 -- 172.21.15.12:0/2648325723 >> [v2:172.21.15.191:6800/3759890485,v1:172.21.15.191:6801/3759890485] conn(0x55ace8d4e120 msgr2=0x55ace8d50550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:28.913 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.914 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.912+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.915 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 zone default found 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 4 Realm: () 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 zonegroup default 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.913+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.916 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 rados->read ofs=0 len=0 2023-12-03T11:39:28.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:28.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 started sync module instance, tier type = 2023-12-03T11:39:28.917 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.914+0000 7f158053b800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:39:28.922 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.921+0000 7f157ca41640 0 -- 172.21.15.12:0/2648325723 >> [v2:172.21.15.12:6824/518024637,v1:172.21.15.12:6825/518024637] conn(0x55ace8d561a0 msgr2=0x55ace8d58610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=0 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=1 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=2 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=3 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=4 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=5 2023-12-03T11:39:28.925 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.924+0000 7f158053b800 20 add_watcher() i=6 2023-12-03T11:39:28.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.925+0000 7f158053b800 20 add_watcher() i=7 2023-12-03T11:39:28.926 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.925+0000 7f158053b800 2 all 8 watchers are set, enabling cache 2023-12-03T11:39:28.927 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.926+0000 7f15427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.927+0000 7f158053b800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:39:28.928 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.927+0000 7f158053b800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:39:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.973+0000 7f14b17fa640 20 reqs_thread_entry: start 2023-12-03T11:39:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.973+0000 7f158053b800 20 init_complete bucket index max shards: 1 2023-12-03T11:39:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.973+0000 7f158053b800 20 Filter name: none 2023-12-03T11:39:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.973+0000 7f14aaffd640 20 reqs_thread_entry: start 2023-12-03T11:39:28.975 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.973+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2023-12-03T11:39:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.975+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:28.976 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.975+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1_9f0c91573c7cf554525d76cb4ebcbe44', time=2023-12-03T13:34:42.418792+0000, chain.objs.size()=5 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2 2023-12-03T11:39:28.978 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2_1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.3 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12164665748435132370', time=2023-12-03T13:36:38.944409+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.2DTFmsUpH4LC7dfKS6aTywsv1qmiao6_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14190868907702440245', time=2023-12-03T13:37:10.128029+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.oo-RKlhlsdEOdRba2xTTkd9AbFifdgN_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16708392959667990896', time=2023-12-03T13:37:11.400202+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.lnsp6VzsB4PY9Ut2Jyu3kSRPA9XTxSO_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18328953813443073376', time=2023-12-03T13:37:11.402660+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.SEHI3mXVNoSQwp1WrWRLDvVz4u4gbFK_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7913172819097422765', time=2023-12-03T13:37:11.978193+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.SbUa1bcckH6qHKBLfrZvow61DXNAbSW_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.976+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7312630474262983182', time=2023-12-03T13:37:43.093723+0000, chain.objs.size()=1 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.977+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.DCfrFwsZHLDMvDjDnoXcwligs_dgoG8_0 2023-12-03T11:39:28.979 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.977+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6068685629097936253', time=2023-12-03T13:37:43.099176+0000, chain.objs.size()=1 2023-12-03T11:39:28.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.977+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.CP-PoexjmEcAFIQpljvwOSeJIXmIao7_0 2023-12-03T11:39:28.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.978+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5924435560478083512', time=2023-12-03T13:37:43.689557+0000, chain.objs.size()=1 2023-12-03T11:39:28.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.978+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.EgMGUNcjiJ5CzD0xWTvnIOQ_onmgVAJ_0 2023-12-03T11:39:28.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.978+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8251359777401269683', time=2023-12-03T13:38:15.260991+0000, chain.objs.size()=1 2023-12-03T11:39:28.980 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.978+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.SOEAnwOCd4DTqivppaG58nrN60J4irW_0 2023-12-03T11:39:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.980+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7423143106751612556', time=2023-12-03T13:38:15.263795+0000, chain.objs.size()=1 2023-12-03T11:39:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.980+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_._1oakyCr3HFhQKVMNbZHU7gpm4G5eEK_0 2023-12-03T11:39:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.980+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3098464038433818444', time=2023-12-03T13:38:15.924323+0000, chain.objs.size()=1 2023-12-03T11:39:28.981 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.980+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.M1M_3PPvMs_yXveahbCJd1RvJ9G6wVW_0 2023-12-03T11:39:28.985 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.984+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:28.988 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.987+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2023-12-03T11:39:28.989 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.988+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10562433295627422172', time=2023-12-03T13:34:42.338565+0000, chain.objs.size()=4 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.1 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.2 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.3 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.8__multipart_foo.2~ndu-X6u2sQ9hUdvdkNBWfzlI1YhfFJW.4 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10575020942548453344', time=2023-12-03T13:36:06.934638+0000, chain.objs.size()=1 2023-12-03T11:39:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.N17sg4UnSS18Z7sr3wHpiDnEgtZZCV__0 2023-12-03T11:39:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12064950063172197923', time=2023-12-03T13:36:37.873214+0000, chain.objs.size()=1 2023-12-03T11:39:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.O7JFhF7zgGMF4lGIlIV08gkc_AS6ZfC_0 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15646495812460774757', time=2023-12-03T13:36:37.874059+0000, chain.objs.size()=1 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.07D1tgg81Uk4H6vkKUHxmgD0R5nQYRw_0 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17029080327624364482', time=2023-12-03T13:37:10.123978+0000, chain.objs.size()=1 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.YlCoaFsRTQ2JN3WnJMwVRtoS6f6PGKc_0 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8752781586658351039', time=2023-12-03T13:37:11.402722+0000, chain.objs.size()=1 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.oyXfpi9XmEwGcPwXir6eO5p6tEN30tl_0 2023-12-03T11:39:28.993 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2505687622178017021', time=2023-12-03T13:37:43.108468+0000, chain.objs.size()=1 2023-12-03T11:39:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.989+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.54QCvOVZVbuQ3RXr5AcO8xDom1Hineg_0 2023-12-03T11:39:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.990+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15467897129080707625', time=2023-12-03T13:38:15.257521+0000, chain.objs.size()=1 2023-12-03T11:39:28.994 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.990+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.5dNnSNBQqZpuTuh4JxIZmf8sv0WZcYL_0 2023-12-03T11:39:28.995 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.994+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:28.998 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.997+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2023-12-03T11:39:28.999 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.998+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.000 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10680517398381153297', time=2023-12-03T13:36:06.267905+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.NbUdbTj1Zxgj8j4MZR1qNYuIaHK7JP-_0 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11445860010329124792', time=2023-12-03T13:36:06.270093+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.OX2EW7ZyPNd7YpEOffpyTg3usQmKUne_0 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7938815215763067557', time=2023-12-03T13:36:38.928292+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.xdiNvhVlJ2mjKPszFkLohRQKxAtFtq9_0 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3721342483941261441', time=2023-12-03T13:36:38.928978+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.PwuWYgftkpUzJ52IY7NDKd6ms0Y-Atp_0 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17749178461568061026', time=2023-12-03T13:37:10.119437+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.nX3X_dpZ9fwReLu1TNlSkLC1kdhq5kz_0 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15087269727064244976', time=2023-12-03T13:37:11.965498+0000, chain.objs.size()=1 2023-12-03T11:39:29.001 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.hz0Qdw3fjZ7Qj-U5IRZlN6E4NaRAXi5_0 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5800909232011017325', time=2023-12-03T13:37:11.968281+0000, chain.objs.size()=1 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.fxtvfvO0zhEuGzFV0bkkJxfBpArd8ZN_0 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11257312298802275671', time=2023-12-03T13:38:15.249359+0000, chain.objs.size()=1 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.Trh8Jgx75hsVYmZbue05zvZrJlDEzMb_0 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9858936010206565324', time=2023-12-03T13:38:15.257419+0000, chain.objs.size()=1 2023-12-03T11:39:29.002 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:28.999+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.vWzoYz2bw_1K_aPa8mweNRT9LhsY7x2_0 2023-12-03T11:39:29.005 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.003+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.007 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.006+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2023-12-03T11:39:29.008 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.007+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.009 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14319456499597463394', time=2023-12-03T13:36:06.258081+0000, chain.objs.size()=1 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.CRavsSTer9D9fEbztu2HKhTAif_k_pO_0 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15581395605396864458', time=2023-12-03T13:36:06.933305+0000, chain.objs.size()=1 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.1N6sHbR6fWSgswJp9KgmDpwrl35SCvQ_0 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5160972571924206086', time=2023-12-03T13:36:38.928526+0000, chain.objs.size()=1 2023-12-03T11:39:29.010 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.NTeDgiW0HR369sxJKCLWclfEsy__4R6_0 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13381974769667394036', time=2023-12-03T13:37:11.391019+0000, chain.objs.size()=1 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.c6N01_RvztY8oNPqnNGB9Mv9jK6nqoN_0 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6428604939797413359', time=2023-12-03T13:37:11.971344+0000, chain.objs.size()=1 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.wXCGRnz3VdJQd1hmdmUQByqr3_08D_C_0 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1463949980854549410', time=2023-12-03T13:37:43.098969+0000, chain.objs.size()=1 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.jphtf-ZBsOvYaWoFaIg-_QdVbsBDh9o_0 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2180112994165424863', time=2023-12-03T13:37:44.173716+0000, chain.objs.size()=1 2023-12-03T11:39:29.011 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.008+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.SW3cS-A8ysKqXvUWy4rC4q6AjbM9mlV_0 2023-12-03T11:39:29.013 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.011+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.018 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.017+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2023-12-03T11:39:29.019 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.018+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.020 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2023-12-03T11:39:29.021 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16126398870454817182', time=2023-12-03T13:34:42.234532+0000, chain.objs.size()=1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.goh4Tz9xQljybkUMgEfNwI81uf8w6dU_0 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16701092478190069115', time=2023-12-03T13:34:42.248104+0000, chain.objs.size()=1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.nHk-WsKr1VbSdaR9ZyUp6pd9FZrj0m__0 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1_8288a923b35e20268a970b534527cd21', time=2023-12-03T13:34:42.358115+0000, chain.objs.size()=2 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_0 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1_f978228b1d79971080fe4fec24e641d1', time=2023-12-03T13:34:42.378192+0000, chain.objs.size()=2 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_0 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2326866221525969259', time=2023-12-03T13:36:38.927667+0000, chain.objs.size()=1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.gnWbYQBNH5ASAboIQaaS0OnSOh8w1VD_0 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.019+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16429850232269318656', time=2023-12-03T13:36:38.930949+0000, chain.objs.size()=1 2023-12-03T11:39:29.022 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.020+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.xoQd1PpOPdjredNVZJysxRq0iJpdF3U_0 2023-12-03T11:39:29.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.020+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10652708519505869990', time=2023-12-03T13:36:38.937191+0000, chain.objs.size()=1 2023-12-03T11:39:29.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.020+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.HrrSxbxmwJf6LPOVkcy2qxPPmspe8vX_0 2023-12-03T11:39:29.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.020+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14739160172049174787', time=2023-12-03T13:37:44.164546+0000, chain.objs.size()=1 2023-12-03T11:39:29.023 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.020+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.RLCFzhraa3SqsYF5Pv_qFmzJbyBb8An_0 2023-12-03T11:39:29.024 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.023+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.027 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.025+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2023-12-03T11:39:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.027+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.028 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.027+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:39:29.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17566623543330718907', time=2023-12-03T13:36:37.867821+0000, chain.objs.size()=1 2023-12-03T11:39:29.029 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.Y6xIs_XP18SsuFziGTjZ0Hao3w9m1Zz_0 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.644965522027916531', time=2023-12-03T13:36:38.933206+0000, chain.objs.size()=1 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.3kk9alLXgc4s_S3oiX7Zvquv_HGyIpF_0 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11885258682169083554', time=2023-12-03T13:37:11.397381+0000, chain.objs.size()=1 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.AwHesxhrvLeCpLUzmvlifm_VvOHBcwe_0 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.80795718788242002', time=2023-12-03T13:37:44.168775+0000, chain.objs.size()=1 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.dKe2VnJRZVop99l_vEaLKbOc9LTPy_V_0 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16700419434368044873', time=2023-12-03T13:38:15.261054+0000, chain.objs.size()=1 2023-12-03T11:39:29.030 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.028+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.fOh2xyw5GZxC-RfKsddnzwOdQYfQBNI_0 2023-12-03T11:39:29.032 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.031+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.034 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.033+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2023-12-03T11:39:29.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.035+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.036 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.035+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1649857100054216235', time=2023-12-03T13:36:06.256706+0000, chain.objs.size()=1 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.zegXSYa8NqPFnqv4e0OydsABZQff8d4_0 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11065017697204811362', time=2023-12-03T13:37:10.126714+0000, chain.objs.size()=1 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.Zbo7hmZpguWJgbMxZzsg-EtjGDWG9nB_0 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16880924675092904406', time=2023-12-03T13:37:11.389249+0000, chain.objs.size()=1 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.B3-wm7LpWF1CdWw6ZWoIEg1Fcf8RLfl_0 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6244986113328080115', time=2023-12-03T13:37:11.394876+0000, chain.objs.size()=1 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.zCxFJqImq3zJWwAjEvYDEzXGy-lqp7Q_0 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15192816157379986692', time=2023-12-03T13:37:43.102411+0000, chain.objs.size()=1 2023-12-03T11:39:29.040 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.U8TKNYprT55qZO6NQe-RVuBMVlpaNhS_0 2023-12-03T11:39:29.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17016220147863310827', time=2023-12-03T13:37:44.171305+0000, chain.objs.size()=1 2023-12-03T11:39:29.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.w31PWexbUycZZjjpDNTUlT6c7CO_kcs_0 2023-12-03T11:39:29.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1739783739511615834', time=2023-12-03T13:38:26.542658+0000, chain.objs.size()=1 2023-12-03T11:39:29.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.036+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:PrRUY5Gwvaj8KZ4Zcr-O6jCG9U655Us_.cSAsLGUDLR1oUJVAjnWUV_THR_dkXEV_0 2023-12-03T11:39:29.041 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.039+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.043 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.042+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2023-12-03T11:39:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.043+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.044 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.043+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:39:29.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2105355681576311583', time=2023-12-03T13:34:42.252114+0000, chain.objs.size()=1 2023-12-03T11:39:29.045 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.Btk0UtDG5JLyWnyBODQZBAQVcIjb7qx_0 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17156515010278605568', time=2023-12-03T13:34:42.430697+0000, chain.objs.size()=5 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2_1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.3 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10190325541161139776', time=2023-12-03T13:34:42.489590+0000, chain.objs.size()=3 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.2 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-STANDARD.2~GUlPabHtmZwmy_zfzu6LGx3JQgNj9EP.3 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14846886487124882191', time=2023-12-03T13:34:42.591704+0000, chain.objs.size()=1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.SEMWcHTVH9c-gmA3yYa9D8UgTw_SE3q_0 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7203661190220013060', time=2023-12-03T13:36:06.265769+0000, chain.objs.size()=1 2023-12-03T11:39:29.046 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.v6LLsw6Yv141e_-pQV1Ll6ezwx5EBts_0 2023-12-03T11:39:29.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4661345622070838436', time=2023-12-03T13:36:37.875525+0000, chain.objs.size()=1 2023-12-03T11:39:29.047 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.044+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.P5-GCb0Z0qk1Nk4181ZfdpNiWifaluH_0 2023-12-03T11:39:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7843950053266119017', time=2023-12-03T13:36:37.879602+0000, chain.objs.size()=1 2023-12-03T11:39:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.aRcD3XOl9kRsQUWrLuQlBUu6uWHfsDI_0 2023-12-03T11:39:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10322010773609756305', time=2023-12-03T13:37:11.396014+0000, chain.objs.size()=1 2023-12-03T11:39:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.CocIhD1yhLz9Z-rDk-mvKob89eXQzqN_0 2023-12-03T11:39:29.049 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17311762445898963979', time=2023-12-03T13:37:43.107553+0000, chain.objs.size()=1 2023-12-03T11:39:29.050 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.048+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.L_6vJAXiG1_IEGQfvbohp9ihQZlKHLz_0 2023-12-03T11:39:29.054 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.053+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.057 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.056+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2023-12-03T11:39:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.057+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.058 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.057+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_1_b3e931a60336b15e58c434f8cd6d3d02', time=2023-12-03T13:34:42.398485+0000, chain.objs.size()=1 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_0 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1_93efd03e865c66d0bee9d43ee419c831', time=2023-12-03T13:34:42.439273+0000, chain.objs.size()=5 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.1_1 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.2_1 2023-12-03T11:39:29.060 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-LUKEWARM.2~dgNmjXfynyQOr4GzQQ0rR1hcGyvM-Yq.3 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2176561685551083003', time=2023-12-03T13:36:06.858981+0000, chain.objs.size()=1 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.UdwZ5hNx7MpkV7XKLbpbz3GRgzP-l--_0 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10616599106846626926', time=2023-12-03T13:37:10.127373+0000, chain.objs.size()=1 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.hZl2zK2MPkKQq67PJDyXLCf0Eq1qstC_0 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9217057139972424171', time=2023-12-03T13:37:10.128059+0000, chain.objs.size()=1 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.iKcjS8lPFGmPmcn0PFRdxlfLCBD2iFq_0 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14851610812026730747', time=2023-12-03T13:37:11.390087+0000, chain.objs.size()=1 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.CJ4SMblW78iMqExBaqPBxHYXCyM7aBp_0 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11000277245859305788', time=2023-12-03T13:37:44.166280+0000, chain.objs.size()=1 2023-12-03T11:39:29.061 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.058+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.iAUkWiUkur_7_xoKHD6CJfiOdwHdgFJ_0 2023-12-03T11:39:29.062 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.061+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.064+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2023-12-03T11:39:29.066 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.065+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.065+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2043411954430940023', time=2023-12-03T13:34:42.564001+0000, chain.objs.size()=1 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.rzrdPuK7suFB6IAe8G1aL20Bs9SkeEI_0 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3893165672394165790', time=2023-12-03T13:34:42.579483+0000, chain.objs.size()=1 2023-12-03T11:39:29.067 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.2X2k7HOq0Z6tXdzOostShE79p09ga65_0 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3506027523601403910', time=2023-12-03T13:36:37.876896+0000, chain.objs.size()=1 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.QYv_B_5LBf0M-aii8kDOdfEtjpDKy0C_0 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14349133205646869247', time=2023-12-03T13:37:11.960199+0000, chain.objs.size()=1 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_._cq1DX6azxXBo2NqLBCaHLHs7LUErK0_0 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.462542176848328816', time=2023-12-03T13:38:15.258844+0000, chain.objs.size()=1 2023-12-03T11:39:29.068 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.066+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.4braq49a-7gZy48Fvppd1KrrGFd4M2M_0 2023-12-03T11:39:29.069 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.068+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.071+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2023-12-03T11:39:29.073 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.072+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.074 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.072+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=12, truncated=0, next_marker='' 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_0xKGGxSWfhvAtKT5m_jurXGdnIkYTkp', time=2023-12-03T13:34:42.382386+0000, chain.objs.size()=2 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.yvFj7buJ7KMSfXqYQ7AkdsZMJlxOfW5_0 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.yvFj7buJ7KMSfXqYQ7AkdsZMJlxOfW5_1 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_YU750ByhBLheLwkrjXna69x0WwQfhQ_', time=2023-12-03T13:34:42.390214+0000, chain.objs.size()=1 2023-12-03T11:39:29.075 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.qGrMPm1DHF3kob_qiprLbzjfLRUnSvs_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5158658116195853606', time=2023-12-03T13:34:42.513381+0000, chain.objs.size()=3 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.CEXOuDqyR8rbWy1VjrLefWuO6-DTShX_2 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7794822273814953865', time=2023-12-03T13:36:06.263234+0000, chain.objs.size()=1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.7YMAx-m6zQz8btihPeMmVsvcsKqmC89_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6944634607634326472', time=2023-12-03T13:36:06.264920+0000, chain.objs.size()=1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.14pey6Sc_w35h6Al7dt7QrbdE8guGC6_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15622719922256473916', time=2023-12-03T13:36:06.863407+0000, chain.objs.size()=1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.afpNpcQCsSzo5mlBWrq3qpGNw31jZIp_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2956707546407602460', time=2023-12-03T13:36:37.870153+0000, chain.objs.size()=1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.Xy-A2ZgghS_6Ii8kVXk43uiVVP7-rKR_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18207192234275869382', time=2023-12-03T13:36:38.944770+0000, chain.objs.size()=1 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.S5P_Casu-5uxfFh_pXLx52SRSyEhYDL_0 2023-12-03T11:39:29.076 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17462693912418630402', time=2023-12-03T13:37:11.968573+0000, chain.objs.size()=1 2023-12-03T11:39:29.077 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.073+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.hQyir2qKfTXhuYZVXlPtBulaKbTbxVK_0 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.077+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4926300753322480078', time=2023-12-03T13:37:11.978266+0000, chain.objs.size()=1 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.077+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.8c_HRie3j0S3wd4XK7aFItCnuRhyPJV_0 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.077+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10221492717752533856', time=2023-12-03T13:37:43.089723+0000, chain.objs.size()=1 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.078+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.F-4rUp6aSTTpTroQdYYkLGNrVJGdr03_0 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.078+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12189471285649197174', time=2023-12-03T13:38:15.252333+0000, chain.objs.size()=1 2023-12-03T11:39:29.079 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.078+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.1mJ-fXCEQLrL4tP8cRoOfU4WJtaqWVQ_0 2023-12-03T11:39:29.083 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.082+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.086 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.084+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2023-12-03T11:39:29.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.087 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.664415852394519092', time=2023-12-03T13:34:42.238593+0000, chain.objs.size()=1 2023-12-03T11:39:29.089 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.T_2I9F-xCvgOvUjz1P0-O2ddA_dCaQR_0 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.377449586680940161', time=2023-12-03T13:34:42.285028+0000, chain.objs.size()=4 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.1 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.2 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.3 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.10__multipart_foo.2~OdDwqyzbhN4UxoCM1LBtkVYwJeWggyX.4 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_UTu5_AbtjPsxnDoc4o4iVUd5OD8A350', time=2023-12-03T13:34:42.374258+0000, chain.objs.size()=2 2023-12-03T11:39:29.090 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.hB2mqaw_zMsmOyK_lxY373jVIBIjDxf_0 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.hB2mqaw_zMsmOyK_lxY373jVIBIjDxf_1 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4022595765224458958', time=2023-12-03T13:34:42.481369+0000, chain.objs.size()=5 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.1 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.1_1 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.086+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.2 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.087+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.2_1 2023-12-03T11:39:29.091 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.087+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-FROZEN.2~vxLp0CIKYxhR3oQhEkHb23KpUhZJcvW.3 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.555980054709117748', time=2023-12-03T13:36:06.865806+0000, chain.objs.size()=1 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.9LCIcf5XMNjmfic_68FzaVVL3AdfnVA_0 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16310469427939473494', time=2023-12-03T13:36:37.869314+0000, chain.objs.size()=1 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.5XrnzNHBNiKCZLama2I7x5qXHBZze1Q_0 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13627309542204050036', time=2023-12-03T13:36:37.885450+0000, chain.objs.size()=1 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.z7tU0d132EX5RKPIc0uaVS2Mam1Q3dM_0 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5980029959866222845', time=2023-12-03T13:37:11.393939+0000, chain.objs.size()=1 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.HAFHfQ718ZsMslYYQVnPM-DzsktSCl7_0 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2208790455368979874', time=2023-12-03T13:37:11.398703+0000, chain.objs.size()=1 2023-12-03T11:39:29.093 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.O1ge4XOplmewwetn9BHu7BrQWm8XGyc_0 2023-12-03T11:39:29.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8144432010369060747', time=2023-12-03T13:37:11.965219+0000, chain.objs.size()=1 2023-12-03T11:39:29.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.JBOMxz9wS6YKkVVhbzjRShb_4NKhU1n_0 2023-12-03T11:39:29.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9160352772762577769', time=2023-12-03T13:37:43.099565+0000, chain.objs.size()=1 2023-12-03T11:39:29.094 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.091+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.97RJKeurDfyOFgUChPrqZvu8sgxKhJg_0 2023-12-03T11:39:29.097 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.096+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.101 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.100+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2023-12-03T11:39:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.102+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.103 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.102+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.102+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:39:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.102+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14005564630534972286', time=2023-12-03T13:34:42.508853+0000, chain.objs.size()=3 2023-12-03T11:39:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.102+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_0 2023-12-03T11:39:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_1 2023-12-03T11:39:29.104 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.VxFnTaZck8ytDAJh6VXdbuyk84cY9CN_2 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17644749034285527213', time=2023-12-03T13:36:06.262571+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.BUMU_vo3HVILQaa4NoRQOWPTlo9pcnE_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1769910875191070237', time=2023-12-03T13:36:06.934069+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.LaF0uGN2Cq-tPNrpZEvYmsyGowZqeyy_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5852324163869775515', time=2023-12-03T13:36:37.882600+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.579-q6Xz-xV4Q6IDmrdBK56GWNrlBQJ_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4141941374169379903', time=2023-12-03T13:36:38.932999+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.6M5fSul0tXyIYjDLxAN0SZJtv1VlCur_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7799269843890906146', time=2023-12-03T13:37:11.393078+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.ONCpzvTiw1AsK13RxuAUh3dsqYnY2eP_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.799313324610953311', time=2023-12-03T13:37:43.104408+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.-VEuWOroJ44tRzeWTMZWG2t07JjlC1I_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12246173256248554834', time=2023-12-03T13:37:44.160206+0000, chain.objs.size()=1 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.1TkXBDPe6xrTQyw9ZAxLqjwW5ELknst_0 2023-12-03T11:39:29.105 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15187059842545379950', time=2023-12-03T13:38:15.924996+0000, chain.objs.size()=1 2023-12-03T11:39:29.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.PfwTbQ97zc-2sMh3Y9zXSXx94z93wKF_0 2023-12-03T11:39:29.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16014576554749852192', time=2023-12-03T13:38:15.925683+0000, chain.objs.size()=1 2023-12-03T11:39:29.106 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.103+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.fOkNU7Z4Ia-K4c6MTi9_wEPlEIki0WY_0 2023-12-03T11:39:29.111 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.110+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.114 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.113+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2023-12-03T11:39:29.115 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.114+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.116 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9707980856756902311', time=2023-12-03T13:34:42.260590+0000, chain.objs.size()=1 2023-12-03T11:39:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.piD16c6KKpEYJ5XUG6P4_4IhG5jCewL_0 2023-12-03T11:39:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3646708246750626850', time=2023-12-03T13:34:42.485417+0000, chain.objs.size()=5 2023-12-03T11:39:29.117 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.1_1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.2 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__shadow_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.2_1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.13__multipart_foo-LUKEWARM.2~-4PIWZf5-zYSjCiHYXitzINeN9B_van.3 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7338833774424480203', time=2023-12-03T13:36:06.851712+0000, chain.objs.size()=1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.UpA7J5yYZPMEpnN0a4o0vdbHpVcqReB_0 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8344995654803825215', time=2023-12-03T13:36:06.854096+0000, chain.objs.size()=1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.ayE_3X4chqcNm4NARXd-8xCtKgQ8aYW_0 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14738157907817703833', time=2023-12-03T13:36:07.462093+0000, chain.objs.size()=1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.4OQxDrLJEROZIHIC-kiETXY98CdKF3i_0 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7668234537421920508', time=2023-12-03T13:37:11.391447+0000, chain.objs.size()=1 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.4m5NfUfOOoESOugIrFZkkBAz7LagAr0_0 2023-12-03T11:39:29.118 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6616769406245902841', time=2023-12-03T13:37:11.393235+0000, chain.objs.size()=1 2023-12-03T11:39:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.115+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.Hm31igLY_4pmgmG_-IcPM_JHjoYVwob_0 2023-12-03T11:39:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.116+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9717240202248150214', time=2023-12-03T13:37:11.395692+0000, chain.objs.size()=1 2023-12-03T11:39:29.119 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.116+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.il9zyIIgTDP69mpp2SGkLM6yKQk6kAy_0 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.119+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16444207943553850002', time=2023-12-03T13:37:43.095356+0000, chain.objs.size()=1 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.119+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.10_HmGkAEzzhc-IyLN07u4Ph66uq2L0_0 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.120+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8383260030480276902', time=2023-12-03T13:37:44.176722+0000, chain.objs.size()=1 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.120+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.NjBt1n1bFVohIbVIfCScPY1A967N2jb_0 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.120+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.519253038099561896', time=2023-12-03T13:38:15.922884+0000, chain.objs.size()=1 2023-12-03T11:39:29.121 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.120+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.Q2PNyH-0ZnVr7PTrbXD1YD1cDWBeqEL_0 2023-12-03T11:39:29.125 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.124+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.128 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.127+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2023-12-03T11:39:29.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.128+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.129 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.128+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:39:29.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16799117797126315124', time=2023-12-03T13:36:05.988879+0000, chain.objs.size()=1 2023-12-03T11:39:29.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.yFfaEYZRSMwUhwU4um6F0ED5hG5XvND_0 2023-12-03T11:39:29.130 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10666574996504245333', time=2023-12-03T13:37:10.133896+0000, chain.objs.size()=1 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.EJl9hdFyG6twQBT3hdhzHvG1jYHjvYX_0 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6946425588652222060', time=2023-12-03T13:37:43.099584+0000, chain.objs.size()=1 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.yXVebrZB9CkgH9OLVlBVpOdyU_ZH-_L_0 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12677803360231801893', time=2023-12-03T13:38:15.250642+0000, chain.objs.size()=1 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.VszUlySEqug60VcW_w0SwycsPc7WI-h_0 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16993205087004535156', time=2023-12-03T13:38:15.265539+0000, chain.objs.size()=1 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.yK8EKzB37q5VXJxzHZhXturhYpt_P9b_0 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3327338786937340676', time=2023-12-03T13:38:26.542131+0000, chain.objs.size()=1 2023-12-03T11:39:29.131 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.129+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:Pdx8Vj9T5Zo3oZaYXZNd14ZphGiNYvb_.tE16vvkP0Dhicow0HO99KtM1nVwa90s_0 2023-12-03T11:39:29.133 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.132+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.136 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.135+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2023-12-03T11:39:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.136+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.137 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.136+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.138 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9004487932909448699', time=2023-12-03T13:34:42.555675+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.ydjS7M4OLoX-fecWrekYvIartBD3wpt_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2846600123631982815', time=2023-12-03T13:36:06.258329+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.rZHhI5AVAyapaB23XrSEFVR8Qd5rbIx_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2617505752404615920', time=2023-12-03T13:36:37.877486+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.pKJ0x6jTYBbj_ZCNEbjWVW3QvCue7Jl_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14412511551142464216', time=2023-12-03T13:36:38.933943+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.STPCZUm5BudJHetS16kOFD3IzluTq1F_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14900416348297404178', time=2023-12-03T13:37:11.958493+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.N8tgEdcO6x_ZTkaxxNNA00SNOlav0pl_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11629134738505856134', time=2023-12-03T13:37:44.166129+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.crexUfAS7SyXzpaNGsNVJZdBM-llhK5_0 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5182819431191490687', time=2023-12-03T13:38:15.924945+0000, chain.objs.size()=1 2023-12-03T11:39:29.139 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.137+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.gPB2d55hhF0P3UU9a1yYv33PEP3ne-C_0 2023-12-03T11:39:29.142 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.140+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.144 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.143+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2023-12-03T11:39:29.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.144+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.145 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.144+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.94119380687097541', time=2023-12-03T13:34:42.265135+0000, chain.objs.size()=1 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.x9Z5ci9O6fkApiQxYkQBPY5iruDf95e_0 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_fd_pJEWyfYvVj4ZZI8M9_-vveYFSElp', time=2023-12-03T13:34:42.361870+0000, chain.objs.size()=2 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.t4n29GILAyGKwonzgckkVRFb6jmwJ5v_0 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.t4n29GILAyGKwonzgckkVRFb6jmwJ5v_1 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_SRv3rH0WG6BDHut34XtvK6V5qzTwCBE', time=2023-12-03T13:34:42.394110+0000, chain.objs.size()=1 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.qXqEb2qIH0ywGu17MKGQc7tM95DgYfz_0 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10834019392878205632', time=2023-12-03T13:36:06.266821+0000, chain.objs.size()=1 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.VXg-EcnJUiJmIhujaDIKjaOMVL0jexq_0 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11470581200167897115', time=2023-12-03T13:36:06.851612+0000, chain.objs.size()=1 2023-12-03T11:39:29.147 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.gFFI12e1xtUVxpVkYNIe9a4PFhzS-Tn_0 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3704749478090059295', time=2023-12-03T13:36:06.852271+0000, chain.objs.size()=1 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.u7sLARLTCth9aMZhvPcmfsb5IvZevG0_0 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10080855696237546272', time=2023-12-03T13:36:06.863755+0000, chain.objs.size()=1 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.BGIpElEKVcozqQnlFQ_34Z8319rKMQq_0 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2360752122953527134', time=2023-12-03T13:36:38.937299+0000, chain.objs.size()=1 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.pLUfwdr589dSf038TbrLk6VZ4wYOBqX_0 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1434034103384675235', time=2023-12-03T13:37:43.091389+0000, chain.objs.size()=1 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.MBQBvD4tC_9IuEnAnNfvk9uoGvFHIfG_0 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5071739487065657332', time=2023-12-03T13:38:15.261188+0000, chain.objs.size()=1 2023-12-03T11:39:29.148 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.145+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.hSicv6nj3pdfeWlHL9bVNE0flH_dd7d_0 2023-12-03T11:39:29.151 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.150+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.153 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.152+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2023-12-03T11:39:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.153+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.154 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.153+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=10, truncated=0, next_marker='' 2023-12-03T11:39:29.155 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_O6nk-x8gpx308ejl5J1V6M5S-CowKAT', time=2023-12-03T13:34:42.462080+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.31erVeHLCTZZnojAZR1pWoXkc-wXsQb_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1388340660345660680', time=2023-12-03T13:36:37.880618+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.FbzekUMeGlp420xojhDTtGlcsjwhUC2_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.618192612403714013', time=2023-12-03T13:36:38.925780+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.L9wCCXFmScTSMEUsVQSnXm3cu36-3ew_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10094815336100573661', time=2023-12-03T13:37:10.123883+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.quWN4UkRXzsIxgx1kpxIXPkzsYjY_pS_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11428730331229698422', time=2023-12-03T13:37:10.129953+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.o1oBHBeKQNHRSXQ1evY5L8qpZbYKVHg_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4449943012416498836', time=2023-12-03T13:37:11.391300+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.Xpx-z6Sx0ucGgR65DSCwaS2SAoGXhuW_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5380870951071189031', time=2023-12-03T13:37:11.959037+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.v8NA2cJ1lTppuMAtFENX7LAv9lphRtn_0 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6350361289072680087', time=2023-12-03T13:37:11.975667+0000, chain.objs.size()=1 2023-12-03T11:39:29.156 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.zlq5HG60DhY4_Pgr4Ij2NnB0AT46ZRo_0 2023-12-03T11:39:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11767567282085707080', time=2023-12-03T13:37:43.094184+0000, chain.objs.size()=1 2023-12-03T11:39:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.Wn8oY4YdcMzN4nrdr2ok6abMHA7ScOz_0 2023-12-03T11:39:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11637451657718255475', time=2023-12-03T13:37:43.098590+0000, chain.objs.size()=1 2023-12-03T11:39:29.157 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.154+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.AXSqoSkcvrFCWg4J9paqHNUSb1LIDL3_0 2023-12-03T11:39:29.159 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.158+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.161 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.160+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2023-12-03T11:39:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.162+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.163 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.162+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.893260467171684828', time=2023-12-03T13:34:42.414815+0000, chain.objs.size()=5 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.1_1 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2 2023-12-03T11:39:29.164 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.2_1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-FROZEN.2~E9lUQoDEjQHYHadwh6y2LYbY8r6Qv0Y.3 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15089048014206979687', time=2023-12-03T13:36:06.256347+0000, chain.objs.size()=1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.11CcGv30lOR-i0IxZdcLjTDNf-pMCsU_0 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12850987046056777550', time=2023-12-03T13:36:06.264651+0000, chain.objs.size()=1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.Jh8y6tetdo-Lxlr1wWdPxBLh9U_yPPF_0 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13010998844577430249', time=2023-12-03T13:37:10.125984+0000, chain.objs.size()=1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.I43ZiQpKg8lqEdT_asBIQw8MiDKXXMw_0 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2471265839929885273', time=2023-12-03T13:37:11.964483+0000, chain.objs.size()=1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.FTCF35u_N2sNUVBRXjLtZpFBD4NQi-M_0 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18262877606078215507', time=2023-12-03T13:37:44.182320+0000, chain.objs.size()=1 2023-12-03T11:39:29.165 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.163+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.FEdVKwZnDk4Zah2uT5BrUNK9zx4KSyZ_0 2023-12-03T11:39:29.167 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.166+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.169 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.168+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2023-12-03T11:39:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.169+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.171 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_JK6FWFvb6xQMUICFHBJPUNRR3oPAiPR', time=2023-12-03T13:34:42.443167+0000, chain.objs.size()=3 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_0 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_1 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.X0J9xRQyXPM2iDIjPfYNVkauim0sh9W_2 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16022695893183056462', time=2023-12-03T13:36:06.257473+0000, chain.objs.size()=1 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.khlJT3uGyF43QjzAnSZrp1CWTRbaJfp_0 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13969284115777136447', time=2023-12-03T13:36:06.259690+0000, chain.objs.size()=1 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.JwNUYQf2gPN-sal5zG1ct2lOvccJLAM_0 2023-12-03T11:39:29.172 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13283336616379341352', time=2023-12-03T13:37:11.968000+0000, chain.objs.size()=1 2023-12-03T11:39:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.kYBXlJ8AmTnELjzj5iWuyPjkLTiZ-iF_0 2023-12-03T11:39:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6742269510969440782', time=2023-12-03T13:37:44.164760+0000, chain.objs.size()=1 2023-12-03T11:39:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.gl1npdnlb_czJdXcgRV2K9-IDoeGJrJ_0 2023-12-03T11:39:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9085380884075701182', time=2023-12-03T13:38:15.248236+0000, chain.objs.size()=1 2023-12-03T11:39:29.173 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.170+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.0byj87eoETkpa-sh4SE2xsJucNAuqMF_0 2023-12-03T11:39:29.176 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.175+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.178 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.177+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2023-12-03T11:39:29.179 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.178+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.180 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=4, truncated=0, next_marker='' 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_q48Z4DHuzOWBmcZoZnQqv7jqYjPS9gs', time=2023-12-03T13:34:42.422681+0000, chain.objs.size()=3 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_0 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_1 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.K1pVDsRaCfF0jSphrnO1EHl-hj5cweI_2 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5449217867432671317', time=2023-12-03T13:36:06.850227+0000, chain.objs.size()=1 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.0Vem0KBbIZDzXof2FKTrU1aX-pLnjLe_0 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11911364792902579154', time=2023-12-03T13:36:38.661703+0000, chain.objs.size()=1 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.mH21Uk3wZlt68H5hMaUMOFLDBUznPAw_0 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17213601510161593761', time=2023-12-03T13:37:11.388987+0000, chain.objs.size()=1 2023-12-03T11:39:29.181 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.179+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.uj5eZCYeaSe9cGjyeE9yzgfYzm0wakL_0 2023-12-03T11:39:29.184 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.183+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.186 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.185+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2023-12-03T11:39:29.187 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.186+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.188 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=8, truncated=0, next_marker='' 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16996394077932431862', time=2023-12-03T13:34:42.551553+0000, chain.objs.size()=1 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.GG6UUn4-wsGBEEcK1x88pdCpCwCvqIB_0 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13428510153144060377', time=2023-12-03T13:36:06.265736+0000, chain.objs.size()=1 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.Tu7zlvpoCXzuzn4x5bjUxaaAuInVKVC_0 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17940687737282315988', time=2023-12-03T13:36:37.869430+0000, chain.objs.size()=1 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.SxXbgsf-wAiMafkxxoj3vVLO-Nlxq_t_0 2023-12-03T11:39:29.189 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10271932433148026627', time=2023-12-03T13:37:10.121478+0000, chain.objs.size()=1 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.-C9FOBcEsdQrL79IgSOCQlcasB0EPPY_0 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7645868485560082297', time=2023-12-03T13:37:10.133662+0000, chain.objs.size()=1 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.lOZw21QMjVMKL_E9k6C_ImpvIPLYUKC_0 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3256127010481351592', time=2023-12-03T13:37:10.136879+0000, chain.objs.size()=1 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.CV4b91mYQ6TZvOXCekC9xRAr9qFDqGs_0 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3909808210343316381', time=2023-12-03T13:37:10.919459+0000, chain.objs.size()=1 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.1rtqm08cVI-kdSZ4cKZz8M_qw6IWGYt_0 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14414590655853671067', time=2023-12-03T13:38:15.250539+0000, chain.objs.size()=1 2023-12-03T11:39:29.190 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.187+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.Ez16yXk40HS8QOILOwczvOLXYbPXd05_0 2023-12-03T11:39:29.192 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.191+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.197 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.196+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2023-12-03T11:39:29.199 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.198+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.530211647011875016', time=2023-12-03T13:34:42.299636+0000, chain.objs.size()=4 2023-12-03T11:39:29.202 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.1 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.2 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.3 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.11__multipart_foo.2~IOBiHt7d6cy1BdOkDTNEe1k9WJ1jZtk.4 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6803613428353269918', time=2023-12-03T13:36:06.857294+0000, chain.objs.size()=1 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.PuUr0xt6kH6lJdhU6OXaBtp3a6qJjmi_0 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7887378166797143920', time=2023-12-03T13:36:06.935968+0000, chain.objs.size()=1 2023-12-03T11:39:29.203 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.ZINJRt6Ua7cV1M8ymEnpLjUfnDjA30__0 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.16596129865200013138', time=2023-12-03T13:37:10.123790+0000, chain.objs.size()=1 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.0FuF9Igtxxr-_25haLNzHQ64e1Shypx_0 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3176030973110752844', time=2023-12-03T13:37:10.125938+0000, chain.objs.size()=1 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.yHtPH9LY1nH8-9O4Q-Np5Vp2mduR_Tc_0 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1732716596842915221', time=2023-12-03T13:37:43.100361+0000, chain.objs.size()=1 2023-12-03T11:39:29.204 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.Wr3inNdkP-dbYAeH6vQ8bgO11IQWbnP_0 2023-12-03T11:39:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17708556349439127097', time=2023-12-03T13:37:43.102665+0000, chain.objs.size()=1 2023-12-03T11:39:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.199+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.LVjeLgumVP5oZw7wERpRkZ4xz4ZrWNf_0 2023-12-03T11:39:29.205 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.204+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.212 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.210+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2023-12-03T11:39:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.212+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.213 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.212+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.215 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17493518590605276871', time=2023-12-03T13:34:42.256363+0000, chain.objs.size()=1 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.urO1Dy28044OoxneMQJq0QmYSlQyq84_0 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_a47qrs15368aRKAANTnVoyZ_WD1PI4t', time=2023-12-03T13:34:42.434822+0000, chain.objs.size()=3 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_0 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_1 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.pOPe6BzlQGR7hub5VLhz7arSGaRRnpd_2 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2_5ec11d90dc305c0665ec50081de57f71', time=2023-12-03T13:34:42.466191+0000, chain.objs.size()=3 2023-12-03T11:39:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.1 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.3 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7325489356933763225', time=2023-12-03T13:34:42.559728+0000, chain.objs.size()=1 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.6__shadow_.5zF0ArcA0bXqB0RmQoYHXxDA_DZqbib_0 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12218067135393036826', time=2023-12-03T13:36:06.854522+0000, chain.objs.size()=1 2023-12-03T11:39:29.217 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.Q-Xq4_7_dCthiEAGvdhBDASNdiKOQ05_0 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1256092488570745672', time=2023-12-03T13:36:06.860300+0000, chain.objs.size()=1 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.8LPlfhV0kXKQxyWi1gd2aq60Quqn28s_0 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2802272271836114084', time=2023-12-03T13:36:06.862764+0000, chain.objs.size()=1 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.cApf52DfnT2o0hmVxpyV9-VYLlsHF8-_0 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4590109965409811991', time=2023-12-03T13:37:11.968778+0000, chain.objs.size()=1 2023-12-03T11:39:29.218 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.213+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.vGYwHRUkaTqvV7dyIWJr9UBMsroEJEE_0 2023-12-03T11:39:29.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.217+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18103031676197568428', time=2023-12-03T13:37:11.973727+0000, chain.objs.size()=1 2023-12-03T11:39:29.219 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.217+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.LuHr_LzrcuthH0nOXSze2onYYrr-vlZ_0 2023-12-03T11:39:29.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.217+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13525462711851447995', time=2023-12-03T13:37:43.108284+0000, chain.objs.size()=1 2023-12-03T11:39:29.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.217+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.qvGc99vwlADW61Gr3mh7JKEZZdmnfpr_0 2023-12-03T11:39:29.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.218+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6276365531684261897', time=2023-12-03T13:37:44.154925+0000, chain.objs.size()=1 2023-12-03T11:39:29.220 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.218+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.dgT0f121-SVCIzE3eOKqWuLfxi2gT_3_0 2023-12-03T11:39:29.224 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.223+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.228 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.227+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2023-12-03T11:39:29.229 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.228+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.230 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.228+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=9, truncated=0, next_marker='' 2023-12-03T11:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17812713888131807939', time=2023-12-03T13:37:10.126094+0000, chain.objs.size()=1 2023-12-03T11:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.hHHJR9R9zr7I0bdW5Qb4Tv8epD4DQRx_0 2023-12-03T11:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1580402210898947717', time=2023-12-03T13:37:11.388250+0000, chain.objs.size()=1 2023-12-03T11:39:29.231 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.cvhi8JL4tXnbWD28A3cWNat4iOEhvcE_0 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9362258136825469320', time=2023-12-03T13:37:44.161420+0000, chain.objs.size()=1 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.35i-DpGKtcsJEhfYwEOmeUHqptvWhUW_0 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5881293575612449035', time=2023-12-03T13:37:44.164234+0000, chain.objs.size()=1 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.XvcTI3AEmU-zHq11GIZKpqv4Dv9tqM5_0 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6447128022418369162', time=2023-12-03T13:37:44.168255+0000, chain.objs.size()=1 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.TVYigHa1_KkQOvK9PqqrpnwqWSRHKqp_0 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17898003088408186909', time=2023-12-03T13:37:44.169238+0000, chain.objs.size()=1 2023-12-03T11:39:29.232 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.tVQllFARKpnbe52EwI54-utwfbWxvbp_0 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9133979096357624281', time=2023-12-03T13:38:15.256386+0000, chain.objs.size()=1 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.pzFIpmagKsJst3L9HaAbLjiv1vIbUGd_0 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.17016792826104146855', time=2023-12-03T13:38:15.258022+0000, chain.objs.size()=1 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.io3wO2c9B4s9kSuYxvjngJi1Jb-mBD1_0 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4799341106358461210', time=2023-12-03T13:38:15.260922+0000, chain.objs.size()=1 2023-12-03T11:39:29.233 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.229+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.vPDwahhfqRm4KdPFCE0AAvtIj772w1u_0 2023-12-03T11:39:29.235 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.234+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.237 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.236+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2023-12-03T11:39:29.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.238+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.239 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.238+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10713006005016377709', time=2023-12-03T13:34:42.243838+0000, chain.objs.size()=1 2023-12-03T11:39:29.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.l-yoas34foiVdrDhGQQDauGgYPKQ9y2_0 2023-12-03T11:39:29.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10670431984628084925', time=2023-12-03T13:34:42.353920+0000, chain.objs.size()=2 2023-12-03T11:39:29.241 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_0 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.eVEQ1eK9vZSavDLeoGFrY9a9G7Si-BZ_1 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_3nDu7fnbakQMRNXPQAOqWdyeleq8Ws6', time=2023-12-03T13:34:42.366151+0000, chain.objs.size()=2 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.izHSb4CfBa3z3T7rXAi92UvyW5jotYr_0 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.izHSb4CfBa3z3T7rXAi92UvyW5jotYr_1 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.398216055650256862', time=2023-12-03T13:34:42.386226+0000, chain.objs.size()=1 2023-12-03T11:39:29.242 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.1lJq67pA80KcQVfw62KD_t1hGvqkseS_0 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_6Q0Ha0Zene4pBzxsDkKp-XcJUm2WdLP', time=2023-12-03T13:34:42.426670+0000, chain.objs.size()=3 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_0 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_1 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.-5yorkU4fIdUmibozOFoaXao2_jQ-Jw_2 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15041404514053996763', time=2023-12-03T13:36:06.258950+0000, chain.objs.size()=1 2023-12-03T11:39:29.243 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.KbnUSdhf2XR_en30Jajh1CiRxcQ7o3l_0 2023-12-03T11:39:29.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14235569020111851520', time=2023-12-03T13:36:37.884856+0000, chain.objs.size()=1 2023-12-03T11:39:29.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.XMuV8PazLUHBhn__s4VPBeG-0Vi1WJW_0 2023-12-03T11:39:29.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18406679873608590976', time=2023-12-03T13:36:38.943828+0000, chain.objs.size()=1 2023-12-03T11:39:29.244 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.239+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.AK0CzhdMgkgVde4Gnb-DfQDdgIQr_6O_0 2023-12-03T11:39:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12477660988813435594', time=2023-12-03T13:37:11.393394+0000, chain.objs.size()=1 2023-12-03T11:39:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.23__shadow_.QxHaUbh3GSGllKEO8PaiBr_QtTuoC39_0 2023-12-03T11:39:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7277818012289906703', time=2023-12-03T13:37:43.093619+0000, chain.objs.size()=1 2023-12-03T11:39:29.245 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.CF0oUjDaa8UD6ta1oRntmMghNFXBtu2_0 2023-12-03T11:39:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1521753269143772649', time=2023-12-03T13:38:14.967439+0000, chain.objs.size()=1 2023-12-03T11:39:29.246 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.242+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.ul1JAHX4cccO6zQkottF5DRBUdtl6yH_0 2023-12-03T11:39:29.247 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.246+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.249 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.248+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2023-12-03T11:39:29.251 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.250+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.250+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.252 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13851552007675172905', time=2023-12-03T13:34:42.583550+0000, chain.objs.size()=1 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.LjPJUtcZix2h9Qif6rQ6Q6P3p2NNwoQ_0 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13485283792276036006', time=2023-12-03T13:36:06.262389+0000, chain.objs.size()=1 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.0jCOQ-i1M7jM8SbW-ND8-wnKUpbJEVi_0 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15289574732371438014', time=2023-12-03T13:36:06.269950+0000, chain.objs.size()=1 2023-12-03T11:39:29.253 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.6YCfJDBnlzEDGDWd0hgLE8JXkm9WRv__0 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15010131618162785140', time=2023-12-03T13:36:06.857642+0000, chain.objs.size()=1 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.eiSfEbLgb2TOBq8kpkBuFYaa3S6BJH9_0 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5063416116761897179', time=2023-12-03T13:36:37.881775+0000, chain.objs.size()=1 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.JqGTMFfXs-UNvqhIkqRNi6UVVwr9GQP_0 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9240706581055085079', time=2023-12-03T13:37:11.986327+0000, chain.objs.size()=1 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.QkyeJveem8W1OV-PcVD_lK_vt9ghkKT_0 2023-12-03T11:39:29.254 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2732954502732286875', time=2023-12-03T13:37:43.091305+0000, chain.objs.size()=1 2023-12-03T11:39:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.251+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.1WpfMesarW5H1hJi_uYK_orAagA1qml_0 2023-12-03T11:39:29.255 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.254+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.258 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.257+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2023-12-03T11:39:29.259 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.258+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.258+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=6, truncated=0, next_marker='' 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5532246340294514608', time=2023-12-03T13:34:42.318468+0000, chain.objs.size()=4 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.1 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.2 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.3 2023-12-03T11:39:29.261 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.9__multipart_foo.2~BGZa09qDFW8nThrM1ijjSpkOcgQKFMr.4 2023-12-03T11:39:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5891688534375813733', time=2023-12-03T13:34:42.587742+0000, chain.objs.size()=1 2023-12-03T11:39:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.7__shadow_.XZPLC2lihL9d0LrHmbKE-PuIJgnPjEj_0 2023-12-03T11:39:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4200834969913662996', time=2023-12-03T13:36:37.872274+0000, chain.objs.size()=1 2023-12-03T11:39:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.KdxNejEy0pH3cUXf10dnSHmze1CxbKY_0 2023-12-03T11:39:29.262 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1242411607151520401', time=2023-12-03T13:36:37.879314+0000, chain.objs.size()=1 2023-12-03T11:39:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.JV7F3-MFdqZHm-tA8W-bs2k0fcJhI76_0 2023-12-03T11:39:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3102934037004501857', time=2023-12-03T13:37:42.605886+0000, chain.objs.size()=1 2023-12-03T11:39:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.25__shadow_.Nh3IPvpnKjd2erDfJcInZO7NNTs6Zf7_0 2023-12-03T11:39:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12095695711194673539', time=2023-12-03T13:38:15.256516+0000, chain.objs.size()=1 2023-12-03T11:39:29.263 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.259+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.oZ3ejCganyasEp60zVwbE9eP5SNUqYs_0 2023-12-03T11:39:29.264 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.263+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.266 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.265+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2023-12-03T11:39:29.267 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.266+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.268 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=5, truncated=0, next_marker='' 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4165055183541485873', time=2023-12-03T13:36:06.935502+0000, chain.objs.size()=1 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.CW9VD999gporZRbAhuqgZI5RhluCu3p_0 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14473072249341028398', time=2023-12-03T13:37:10.127466+0000, chain.objs.size()=1 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.KB1zZB5fMvR10oA7mzaMsHo6tGf4tQx_0 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.1135235739391976375', time=2023-12-03T13:37:11.961631+0000, chain.objs.size()=1 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.P7QjqFn1RBcccoasxt5DGFjMKRyYfRE_0 2023-12-03T11:39:29.269 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8173419100335463363', time=2023-12-03T13:38:15.926447+0000, chain.objs.size()=1 2023-12-03T11:39:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.00fUUx0Jmd344wP75kCA-pD6Naf-Gc4_0 2023-12-03T11:39:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18111578105225898044', time=2023-12-03T13:38:15.927251+0000, chain.objs.size()=1 2023-12-03T11:39:29.270 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.267+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.Z_JxVu51ZvvGTkgVRAppt6TJgsguSS-_0 2023-12-03T11:39:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.269+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.272+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2023-12-03T11:39:29.274 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.273+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.275 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18135875262510206899', time=2023-12-03T13:34:42.269346+0000, chain.objs.size()=1 2023-12-03T11:39:29.276 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.3__shadow_.ZsiZ3VNL7aNpxPx0nBshYajzvAisCbo_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7286486258281449550', time=2023-12-03T13:36:06.261415+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.18__shadow_.WBRShURdrG2QBPNyIHmI3NEYRtQU2h5_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12947184766944903868', time=2023-12-03T13:36:06.853261+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.19__shadow_.kCSlNNIgAJVDxbN5rn9p-JK2hu7BobD_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8336261613693403354', time=2023-12-03T13:36:38.930699+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.46Yj6om2vXNJw1Xh_O05FIJmrCfmqU7_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.14817548549271953894', time=2023-12-03T13:36:38.940829+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.Rn5Kc5kEW6pGnhmYg-0S1OfP2E75xv5_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3211737725140023890', time=2023-12-03T13:36:39.501139+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.qEQO3zro8FSE5c43k9qI2MZj_wo__33_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.7591441329978231996', time=2023-12-03T13:37:11.974665+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.Wy8dgTc0JF1NSgw1S9HZKEJasplpgen_0 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3004754611288231309', time=2023-12-03T13:37:11.983508+0000, chain.objs.size()=1 2023-12-03T11:39:29.277 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.3t0Auam687-SAicEfnVCslUyuRUAwb4_0 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.13614556194148048578', time=2023-12-03T13:37:44.155239+0000, chain.objs.size()=1 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.g66GBLV3s0JFlqrbR9xm95UZzgbnbEF_0 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15049507383306226976', time=2023-12-03T13:37:44.174803+0000, chain.objs.size()=1 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.GQzR6-RYTr5ZobR3vhvInN-7UayF7rS_0 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18044992727651100716', time=2023-12-03T13:38:15.924805+0000, chain.objs.size()=1 2023-12-03T11:39:29.278 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.274+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.16__shadow_.6GWfxzdUhGBfo8pzkFTNKs85UATVkaF_0 2023-12-03T11:39:29.281 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.280+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.287 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.286+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2023-12-03T11:39:29.288 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.287+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=7, truncated=0, next_marker='' 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.2481323200884536106', time=2023-12-03T13:36:37.875577+0000, chain.objs.size()=1 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.1_r22ltNBBWOE3ey5NdUHwDl89pp6eT_0 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8493720139402809996', time=2023-12-03T13:37:10.123964+0000, chain.objs.size()=1 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.uMIkFDixInfUdGi_nZxxsPjeyG4USn9_0 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3120866896305903816', time=2023-12-03T13:37:10.129822+0000, chain.objs.size()=1 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.22__shadow_.1xm3wanE7sfM7kQ1wGh1x1XPlOIxptu_0 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10849656451787690322', time=2023-12-03T13:37:11.964399+0000, chain.objs.size()=1 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.24__shadow_.Bv_88IkUv_OduqRAGxByklIgOU3Dbq7_0 2023-12-03T11:39:29.290 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12212474988378076928', time=2023-12-03T13:37:44.167225+0000, chain.objs.size()=1 2023-12-03T11:39:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.VpKwzrn3XbDpbj1eVq44ES5M5IXtSCs_0 2023-12-03T11:39:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.18092600046716451496', time=2023-12-03T13:37:44.170044+0000, chain.objs.size()=1 2023-12-03T11:39:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.26__shadow_.tSPOAyNi2G0n1FGQy4y3fyCrO2G2rKY_0 2023-12-03T11:39:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12101372261419937696', time=2023-12-03T13:38:26.543086+0000, chain.objs.size()=1 2023-12-03T11:39:29.291 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.288+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:BGG82GfCIZE2zrNSPiR2y6o9OYRUFhG_.eKx8W5v08dX2i2zI_gF2r0NsSb14GF5_0 2023-12-03T11:39:29.294 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.293+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.297 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.296+0000 7f158053b800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2023-12-03T11:39:29.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.297+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.299 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.298+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=11, truncated=0, next_marker='' 2023-12-03T11:39:29.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.4466723495057247882', time=2023-12-03T13:34:42.370177+0000, chain.objs.size()=2 2023-12-03T11:39:29.300 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_0 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.lukewarm:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.14__shadow_.0wtSSAHFS1aDVOra5t-DMqeR6pPuZSn_1 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.15494616533731123648', time=2023-12-03T13:34:42.446997+0000, chain.objs.size()=3 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.1 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.2 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__multipart_foo-STANDARD.2~qDhyUauF3Onfk4Ztw08E__LQsQSTwb5.3 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='_3KzmqdT_CDu3EYh4rRWkrzONeuINiny', time=2023-12-03T13:34:42.450748+0000, chain.objs.size()=1 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.15__shadow_.4pzAWd2e03mhHYFp2VSCO_YTWobL3UC_0 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.11787476041521528810', time=2023-12-03T13:34:42.504668+0000, chain.objs.size()=3 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_0 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_1 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data.frozen:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.12__shadow_.Ol_pGlM3xk6v8HUUo_WpjY9QcE1QzXI_2 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.12068999115146568829', time=2023-12-03T13:36:37.875543+0000, chain.objs.size()=1 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.20__shadow_.CbMmcMp7b5YDjPgvbJtrb0NypFO8TJL_0 2023-12-03T11:39:29.301 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.10301313739772603259', time=2023-12-03T13:36:38.930280+0000, chain.objs.size()=1 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.s3HknRgH-zt9ZYxJxb3SfUkEiInFU6Q_0 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.3819892063573130140', time=2023-12-03T13:36:38.935365+0000, chain.objs.size()=1 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.299+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.SY3GZXuA2ieiLHBDt53-8adj4XowSiB_0 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.301+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.8968927971126142015', time=2023-12-03T13:36:38.941302+0000, chain.objs.size()=1 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.301+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.21__shadow_.11GsPeQOpw3o1zwjKSomffNrIP158cf_0 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.301+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.9757539991739075661', time=2023-12-03T13:38:15.254442+0000, chain.objs.size()=1 2023-12-03T11:39:29.302 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.301+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.k3ANkVQBmbXo369tzxnf8LUGujOOhCL_0 2023-12-03T11:39:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.302+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.6923246364268817257', time=2023-12-03T13:38:15.262693+0000, chain.objs.size()=1 2023-12-03T11:39:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.302+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.27__shadow_.vCSmOO7lfex7y1fK0nh8Heln5Qv05p-_0 2023-12-03T11:39:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.302+0000 7f158053b800 20 garbage collection: RGWGC::process iterating over entry tag='ca5e6929-62e3-48b1-b1be-09f9f5399756.4668.5451379517766290582', time=2023-12-03T13:38:22.882631+0000, chain.objs.size()=1 2023-12-03T11:39:29.303 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.302+0000 7f158053b800 5 garbage collection: RGWGC::process removing default.rgw.buckets.data:ca5e6929-62e3-48b1-b1be-09f9f5399756.4562.17__shadow:08am37tMHKXw9IQPo33bfIvVMvP1pZx_.43-j7nA7FQGsbobsCxpOUxokBDCtnKU_0 2023-12-03T11:39:29.306 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.305+0000 7f158053b800 5 garbage collection: RGWGC::process removing entries, marker: 2023-12-03T11:39:29.309 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.308+0000 7f15427e4640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:29.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.310+0000 7f158053b800 20 remove_watcher() i=0 2023-12-03T11:39:29.311 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.310+0000 7f158053b800 2 removed watcher, disabling cache 2023-12-03T11:39:29.312 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.311+0000 7f158053b800 20 remove_watcher() i=1 2023-12-03T11:39:29.313 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.312+0000 7f158053b800 20 remove_watcher() i=2 2023-12-03T11:39:29.314 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.313+0000 7f158053b800 20 remove_watcher() i=3 2023-12-03T11:39:29.315 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.314+0000 7f158053b800 20 remove_watcher() i=4 2023-12-03T11:39:29.316 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.315+0000 7f158053b800 20 remove_watcher() i=5 2023-12-03T11:39:29.317 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.316+0000 7f158053b800 20 remove_watcher() i=6 2023-12-03T11:39:29.318 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.317+0000 7f158053b800 20 remove_watcher() i=7 2023-12-03T11:39:29.324 DEBUG:teuthology.orchestra.run.smithi012:> radosgw-admin gc process --include-all 2023-12-03T11:39:29.374 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:39:29.374 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:39:29.388 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.387+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 realm 2023-12-03T11:39:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.390 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 4 RGWPeriod::init failed to init realm id : (2) No such file or directory 2023-12-03T11:39:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.389+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.390+0000 7f98b0134800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:29.391 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.390+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.391+0000 7f98b0134800 20 rados_obj.operate() r=0 bl.length=1020 2023-12-03T11:39:29.392 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.391+0000 7f98b0134800 20 searching for the correct realm 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.401+0000 7f98b0134800 20 RGWRados::pool_iterate: got zone_info.ca5e6929-62e3-48b1-b1be-09f9f5399756 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.401+0000 7f98b0134800 20 RGWRados::pool_iterate: got zonegroup_info.65f61d41-48c7-4e91-82a8-7a971a42d3b6 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.401+0000 7f98b0134800 20 RGWRados::pool_iterate: got zone_names.default 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.401+0000 7f98b0134800 20 RGWRados::pool_iterate: got zonegroups_names.default 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.401+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98ac603640 0 -- 172.21.15.12:0/2388942610 >> [v2:172.21.15.12:6800/2316181860,v1:172.21.15.12:6801/2316181860] conn(0x556f30c13d40 msgr2=0x556f30c16130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados_obj.operate() r=0 bl.length=46 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados_obj.operate() r=0 bl.length=454 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 zone default found 2023-12-03T11:39:29.404 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 4 Realm: () 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 4 ZoneGroup: default (65f61d41-48c7-4e91-82a8-7a971a42d3b6) 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 4 Zone: default (ca5e6929-62e3-48b1-b1be-09f9f5399756) 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 10 cannot find current period zonegroup using local zonegroup configuration 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 zonegroup default 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.402+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.405 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 rados->read ofs=0 len=0 2023-12-03T11:39:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 rados_obj.operate() r=-2 bl.length=0 2023-12-03T11:39:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 started sync module instance, tier type = 2023-12-03T11:39:29.406 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.403+0000 7f98b0134800 20 started zone id=ca5e6929-62e3-48b1-b1be-09f9f5399756 (name=default) with tier type = 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.412+0000 7f98abe02640 0 -- 172.21.15.12:0/2388942610 >> [v2:172.21.15.12:6816/1420251316,v1:172.21.15.12:6817/1420251316] conn(0x7f98a0069920 msgr2=0x7f98a005d400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=0 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=1 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=2 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=3 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=4 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=5 2023-12-03T11:39:29.414 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.413+0000 7f98b0134800 20 add_watcher() i=6 2023-12-03T11:39:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.414+0000 7f98b0134800 20 add_watcher() i=7 2023-12-03T11:39:29.415 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.414+0000 7f98b0134800 2 all 8 watchers are set, enabling cache 2023-12-03T11:39:29.416 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.415+0000 7f9871fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:29.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.416+0000 7f98b0134800 20 check_secure_mon_conn(): auth registry supported: methods=[2] modes=[2,1] 2023-12-03T11:39:29.417 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.416+0000 7f98b0134800 20 check_secure_mon_conn(): mode 1 is insecure 2023-12-03T11:39:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.457+0000 7f97e8ff9640 20 reqs_thread_entry: start 2023-12-03T11:39:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.457+0000 7f98b0134800 20 init_complete bucket index max shards: 1 2023-12-03T11:39:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.457+0000 7f98b0134800 20 Filter name: none 2023-12-03T11:39:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.457+0000 7f97de7fc640 20 reqs_thread_entry: start 2023-12-03T11:39:29.459 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.457+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=28, max_secs=3600, expired_only=0 2023-12-03T11:39:29.460 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.459+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.460+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.461 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.460+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.462 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.461+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=29, max_secs=3600, expired_only=0 2023-12-03T11:39:29.463 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.462+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.463+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.464 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.463+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.465 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.464+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=30, max_secs=3600, expired_only=0 2023-12-03T11:39:29.466 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.465+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.466+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.467 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.466+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.468 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.467+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=31, max_secs=3600, expired_only=0 2023-12-03T11:39:29.469 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.468+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.469+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.470 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.469+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.471 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.470+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=0, max_secs=3600, expired_only=0 2023-12-03T11:39:29.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.472+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.473 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.472+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.474 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.473+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.475 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.474+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=1, max_secs=3600, expired_only=0 2023-12-03T11:39:29.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.475+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.476 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.475+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.477 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.476+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.478 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.477+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=2, max_secs=3600, expired_only=0 2023-12-03T11:39:29.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.478+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.478+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.479+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.482 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.480+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=3, max_secs=3600, expired_only=0 2023-12-03T11:39:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.481+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.482+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.482+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.485 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.483+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=4, max_secs=3600, expired_only=0 2023-12-03T11:39:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.485+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.486 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.485+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.487 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.486+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.488 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.487+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=5, max_secs=3600, expired_only=0 2023-12-03T11:39:29.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.488+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.489 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.488+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.490 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.489+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.491 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.490+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=6, max_secs=3600, expired_only=0 2023-12-03T11:39:29.492 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.491+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.492+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.493 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.492+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.494 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.493+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=7, max_secs=3600, expired_only=0 2023-12-03T11:39:29.495 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.494+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.495+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.496 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.495+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.497 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.496+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=8, max_secs=3600, expired_only=0 2023-12-03T11:39:29.499 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.498+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.498+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.500 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.499+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.501 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.500+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=9, max_secs=3600, expired_only=0 2023-12-03T11:39:29.502 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.501+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.502+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.503 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.502+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.504 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.503+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=10, max_secs=3600, expired_only=0 2023-12-03T11:39:29.505 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.504+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.506 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.505+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.505+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.507 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.506+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=11, max_secs=3600, expired_only=0 2023-12-03T11:39:29.508 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.507+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.508+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.509 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.508+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.510 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.509+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=12, max_secs=3600, expired_only=0 2023-12-03T11:39:29.511 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.510+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.511+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.512 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.511+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.513 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.512+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=13, max_secs=3600, expired_only=0 2023-12-03T11:39:29.514 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.513+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.514+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.515 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.514+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.516 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.515+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=14, max_secs=3600, expired_only=0 2023-12-03T11:39:29.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.517+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.518 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.517+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.519 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.518+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.520 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.519+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=15, max_secs=3600, expired_only=0 2023-12-03T11:39:29.521 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.520+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.521+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.522 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.521+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.523 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.522+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=16, max_secs=3600, expired_only=0 2023-12-03T11:39:29.524 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.523+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.524+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.525 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.524+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.526 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.525+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=17, max_secs=3600, expired_only=0 2023-12-03T11:39:29.527 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.526+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.527+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.528 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.527+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.530 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.529+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=18, max_secs=3600, expired_only=0 2023-12-03T11:39:29.531 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.530+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.531+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.532 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.531+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.533 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.532+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=19, max_secs=3600, expired_only=0 2023-12-03T11:39:29.534 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.533+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.533+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.535 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.534+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.536 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.535+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=20, max_secs=3600, expired_only=0 2023-12-03T11:39:29.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.536+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.537 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.536+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.537+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.538 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.537+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=21, max_secs=3600, expired_only=0 2023-12-03T11:39:29.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.539+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.540 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.539+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.540+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.542 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.541+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=22, max_secs=3600, expired_only=0 2023-12-03T11:39:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.542+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.542+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.543 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.542+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.544 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.543+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=23, max_secs=3600, expired_only=0 2023-12-03T11:39:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.545+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.545+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.546 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.545+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.547 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.546+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=24, max_secs=3600, expired_only=0 2023-12-03T11:39:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.548+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.548+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.549 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.548+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.551 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.550+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=25, max_secs=3600, expired_only=0 2023-12-03T11:39:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.551+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.551+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.552 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.551+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.554 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.553+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=26, max_secs=3600, expired_only=0 2023-12-03T11:39:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.554+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.555 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.554+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.554+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.556 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.555+0000 7f98b0134800 20 garbage collection: RGWGC::process entered with GC index_shard=27, max_secs=3600, expired_only=0 2023-12-03T11:39:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.557+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned with returned:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.558 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.557+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_list returned NO non expired entries, so setting cache entry to TRUE 2023-12-03T11:39:29.559 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.558+0000 7f98b0134800 20 garbage collection: RGWGC::process cls_rgw_gc_queue_list_entries returned with return value:0, entries.size=0, truncated=0, next_marker='' 2023-12-03T11:39:29.561 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.559+0000 7f9871fe3640 2 rgw data changes log: RGWDataChangesLog::ChangesRenewThread: start 2023-12-03T11:39:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.561+0000 7f98b0134800 20 remove_watcher() i=0 2023-12-03T11:39:29.563 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.561+0000 7f98b0134800 2 removed watcher, disabling cache 2023-12-03T11:39:29.564 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.563+0000 7f98b0134800 20 remove_watcher() i=1 2023-12-03T11:39:29.565 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.564+0000 7f98b0134800 20 remove_watcher() i=2 2023-12-03T11:39:29.566 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.565+0000 7f98b0134800 20 remove_watcher() i=3 2023-12-03T11:39:29.567 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.566+0000 7f98b0134800 20 remove_watcher() i=4 2023-12-03T11:39:29.568 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.567+0000 7f98b0134800 20 remove_watcher() i=5 2023-12-03T11:39:29.569 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.568+0000 7f98b0134800 20 remove_watcher() i=6 2023-12-03T11:39:29.570 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T11:39:29.569+0000 7f98b0134800 20 remove_watcher() i=7 2023-12-03T11:39:29.575 DEBUG:teuthology.orchestra.run.smithi012:> cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test 2023-12-03T11:39:30.049 INFO:teuthology.orchestra.run.smithi012.stdout:Starting a Gradle Daemon (subsequent builds will be faster) 2023-12-03T11:39:33.850 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :clean UP-TO-DATE 2023-12-03T11:39:52.149 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :compileJava 2023-12-03T11:39:52.149 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :processResources 2023-12-03T11:39:52.149 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :classes 2023-12-03T11:39:52.650 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:52.650 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :compileTestJava 2023-12-03T11:39:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:Note: /home/ubuntu/cephtest/s3-tests-java/src/test/java/BucketTest.java uses or overrides a deprecated API. 2023-12-03T11:39:52.650 INFO:teuthology.orchestra.run.smithi012.stderr:Note: Recompile with -Xlint:deprecation for details. 2023-12-03T11:39:52.651 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:52.651 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :processTestResources NO-SOURCE 2023-12-03T11:39:52.651 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :testClasses 2023-12-03T11:39:58.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:58.749 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :test 2023-12-03T11:39:58.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:58.749 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 STARTED 2023-12-03T11:39:59.249 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:59.249 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadLLAPIAWS4 PASSED 2023-12-03T11:39:59.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:59.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 STARTED 2023-12-03T11:39:59.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:59.549 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testAbortMultipartUploadNotFoundLLAPIAWS4 PASSED 2023-12-03T11:39:59.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:39:59.749 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 STARTED 2023-12-03T11:40:00.149 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.149 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadHLAPIAWS4 PASSED 2023-12-03T11:40:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.349 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 STARTED 2023-12-03T11:40:00.349 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.349 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoBucketHLAPIAWS4 PASSED 2023-12-03T11:40:00.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 STARTED 2023-12-03T11:40:00.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testDownloadNoKeyHLAPIAWS4 PASSED 2023-12-03T11:40:00.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:00.649 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 STARTED 2023-12-03T11:40:01.149 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:01.149 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyHLAPIAWS4 PASSED 2023-12-03T11:40:01.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:01.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 STARTED 2023-12-03T11:40:05.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:05.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyMultipleSizesLLAPIAWS4 PASSED 2023-12-03T11:40:05.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:05.749 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 STARTED 2023-12-03T11:40:05.750 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:05.750 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCBucketHLAPIAWS4 PASSED 2023-12-03T11:40:05.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:05.949 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 STARTED 2023-12-03T11:40:05.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:05.949 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartCopyNoSRCKeyHLAPIAWS4 PASSED 2023-12-03T11:40:06.249 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:06.249 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 STARTED 2023-12-03T11:40:06.849 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:06.849 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadHLAPIAWS4 PASSED 2023-12-03T11:40:07.049 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:07.049 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 STARTED 2023-12-03T11:40:07.049 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:07.049 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadNoBucketHLAPIAWS4 PASSED 2023-12-03T11:40:07.149 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:07.149 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 STARTED 2023-12-03T11:40:09.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:09.449 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartDownloadWithPauseHLAPIAWS4 PASSED 2023-12-03T11:40:09.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:09.649 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 STARTED 2023-12-03T11:40:10.049 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:10.049 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadHLAPIAWS4 PASSED 2023-12-03T11:40:10.349 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:10.349 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 STARTED 2023-12-03T11:40:10.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:10.649 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadIncorrectMissingPartLLAPIAWS4 PASSED 2023-12-03T11:40:10.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:10.749 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 STARTED 2023-12-03T11:40:15.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:15.949 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadMultipleSizesLLAPIAWS4 PASSED 2023-12-03T11:40:16.149 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:16.149 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 STARTED 2023-12-03T11:40:17.249 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:17.249 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadOverwriteExistingObjectLLAPIAWS4 PASSED 2023-12-03T11:40:17.349 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:17.349 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 STARTED 2023-12-03T11:40:17.949 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:17.949 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadSmallLLAPIAWS4 PASSED 2023-12-03T11:40:18.149 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:18.149 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 STARTED 2023-12-03T11:40:18.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:18.549 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testMultipartUploadWithPauseAWS4 PASSED 2023-12-03T11:40:18.619 INFO:tasks.rgw.client.0.smithi012.stdout:*** Caught signal (Segmentation fault) ** 2023-12-03T11:40:18.619 INFO:tasks.rgw.client.0.smithi012.stdout: in thread 7f56a6b08640 thread_name:radosgw 2023-12-03T11:40:18.619 INFO:tasks.rgw.client.0.smithi012.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f56d9654db0] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x556967ad366a] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x556967968fb1] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x55696796ca75] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 5: radosgw(+0xc724b0) [0x5569681604b0] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 6: radosgw(+0x3cf496) [0x5569678bd496] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 7: make_fcontext() 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout:2023-12-03T11:40:18.618+0000 7f56a6b08640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: in thread 7f56a6b08640 thread_name:radosgw 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f56d9654db0] 2023-12-03T11:40:18.620 INFO:tasks.rgw.client.0.smithi012.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x556967ad366a] 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x556967968fb1] 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x55696796ca75] 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 5: radosgw(+0xc724b0) [0x5569681604b0] 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 6: radosgw(+0x3cf496) [0x5569678bd496] 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 7: make_fcontext() 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:40:18.621 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 0> 2023-12-03T11:40:18.618+0000 7f56a6b08640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: in thread 7f56a6b08640 thread_name:radosgw 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f56d9654db0] 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x556967ad366a] 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x556967968fb1] 2023-12-03T11:40:18.629 INFO:tasks.rgw.client.0.smithi012.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x55696796ca75] 2023-12-03T11:40:18.630 INFO:tasks.rgw.client.0.smithi012.stdout: 5: radosgw(+0xc724b0) [0x5569681604b0] 2023-12-03T11:40:18.630 INFO:tasks.rgw.client.0.smithi012.stdout: 6: radosgw(+0x3cf496) [0x5569678bd496] 2023-12-03T11:40:18.630 INFO:tasks.rgw.client.0.smithi012.stdout: 7: make_fcontext() 2023-12-03T11:40:18.630 INFO:tasks.rgw.client.0.smithi012.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:40:18.630 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.641 INFO:tasks.rgw.client.0.smithi012.stdout: -101> 2023-12-03T11:40:18.618+0000 7f56a6b08640 -1 *** Caught signal (Segmentation fault) ** 2023-12-03T11:40:18.641 INFO:tasks.rgw.client.0.smithi012.stdout: in thread 7f56a6b08640 thread_name:radosgw 2023-12-03T11:40:18.641 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.641 INFO:tasks.rgw.client.0.smithi012.stdout: ceph version 18.0.0-7537-gbcbf278c (bcbf278cf395873c2a482f5a7b2345faf2160319) reef (dev) 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 1: /lib64/libc.so.6(+0x54db0) [0x7f56d9654db0] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 2: (RGWCompleteMultipart::complete()+0x4a) [0x556967ad366a] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 3: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, rgw::sal::Driver*, bool)+0xb91) [0x556967968fb1] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 4: (process_request(RGWProcessEnv const&, RGWRequest*, std::__cxx11::basic_string, std::allocator > const&, RGWRestfulIO*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string, std::allocator >*, std::chrono::duration >*, int*)+0xf75) [0x55696796ca75] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 5: radosgw(+0xc724b0) [0x5569681604b0] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 6: radosgw(+0x3cf496) [0x5569678bd496] 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 7: make_fcontext() 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-03T11:40:18.642 INFO:tasks.rgw.client.0.smithi012.stdout: 2023-12-03T11:40:18.742 INFO:tasks.rgw.client.0.smithi012.stderr:daemon-helper: command crashed with signal 11 2023-12-03T11:40:18.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:40:18.749 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 STARTED 2023-12-03T11:40:25.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~0s 2023-12-03T11:40:31.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~6s 2023-12-03T11:40:37.821 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~13s 2023-12-03T11:40:44.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~19s 2023-12-03T11:40:50.442 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~25s 2023-12-03T11:40:56.753 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~32s 2023-12-03T11:41:03.063 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~38s 2023-12-03T11:41:09.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~44s 2023-12-03T11:41:15.685 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~50s 2023-12-03T11:41:21.995 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~57s 2023-12-03T11:41:28.307 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~63s 2023-12-03T11:41:34.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~69s 2023-12-03T11:41:40.926 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~76s 2023-12-03T11:41:47.235 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~82s 2023-12-03T11:41:53.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~88s 2023-12-03T11:41:59.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~95s 2023-12-03T11:42:06.160 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~101s 2023-12-03T11:42:12.468 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~107s 2023-12-03T11:42:18.776 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~114s 2023-12-03T11:42:25.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~120s 2023-12-03T11:42:31.394 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~126s 2023-12-03T11:42:37.707 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~133s 2023-12-03T11:42:44.016 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~139s 2023-12-03T11:42:50.324 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~145s 2023-12-03T11:42:56.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~151s 2023-12-03T11:43:02.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~158s 2023-12-03T11:43:09.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~164s 2023-12-03T11:43:15.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~170s 2023-12-03T11:43:21.869 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~177s 2023-12-03T11:43:28.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~183s 2023-12-03T11:43:34.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~189s 2023-12-03T11:43:40.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~196s 2023-12-03T11:43:47.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~202s 2023-12-03T11:43:53.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~208s 2023-12-03T11:43:59.719 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~215s 2023-12-03T11:44:06.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~221s 2023-12-03T11:44:12.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~227s 2023-12-03T11:44:18.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~233s 2023-12-03T11:44:24.953 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~240s 2023-12-03T11:44:31.262 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~246s 2023-12-03T11:44:37.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~252s 2023-12-03T11:44:43.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~259s 2023-12-03T11:44:50.189 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~265s 2023-12-03T11:44:56.497 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~271s 2023-12-03T11:45:02.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~278s 2023-12-03T11:45:09.116 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~284s 2023-12-03T11:45:15.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~290s 2023-12-03T11:45:21.732 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~297s 2023-12-03T11:45:28.040 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.rgw.client.0 is failed for ~303s 2023-12-03T11:45:28.040 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-03T11:45:29.349 INFO:tasks.ceph.osd.0:Sent signal 15 2023-12-03T11:45:29.349 INFO:tasks.ceph.osd.1:Sent signal 15 2023-12-03T11:45:29.350 INFO:tasks.ceph.osd.2:Sent signal 15 2023-12-03T11:45:29.350 INFO:tasks.ceph.osd.3:Sent signal 15 2023-12-03T11:45:29.350 INFO:tasks.ceph.osd.4:Sent signal 15 2023-12-03T11:45:29.350 INFO:tasks.ceph.osd.5:Sent signal 15 2023-12-03T11:45:29.351 INFO:tasks.ceph.osd.6:Sent signal 15 2023-12-03T11:45:29.351 INFO:tasks.ceph.osd.7:Sent signal 15 2023-12-03T11:45:29.351 INFO:tasks.ceph.mon.a:Sent signal 15 2023-12-03T11:45:29.351 INFO:tasks.ceph.mon.c:Sent signal 15 2023-12-03T11:45:29.351 INFO:tasks.ceph.mon.b:Sent signal 15 2023-12-03T11:45:29.352 INFO:tasks.ceph.mgr.y:Sent signal 15 2023-12-03T11:45:29.352 INFO:tasks.ceph.mgr.x:Sent signal 15 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:45:29.350+0000 7f7907003640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 83853) UID: 0 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:45:29.350+0000 7f7907003640 -1 osd.4 253 *** Got signal Terminated *** 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.4.smithi191.stderr:2023-12-03T11:45:29.350+0000 7f7907003640 -1 osd.4 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f5d747f2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 83866) UID: 0 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f5d747f2640 -1 osd.0 253 *** Got signal Terminated *** 2023-12-03T11:45:29.353 INFO:tasks.ceph.osd.0.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f5d747f2640 -1 osd.0 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.354 INFO:tasks.ceph.mon.b.smithi191.stderr:2023-12-03T11:45:29.350+0000 7f45f2521640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i b (PID: 83326) UID: 0 2023-12-03T11:45:29.354 INFO:tasks.ceph.mon.b.smithi191.stderr:2023-12-03T11:45:29.350+0000 7f45f2521640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2023-12-03T11:45:29.354 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f8d65e83640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 83870) UID: 0 2023-12-03T11:45:29.354 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f8d65e83640 -1 osd.1 253 *** Got signal Terminated *** 2023-12-03T11:45:29.354 INFO:tasks.ceph.osd.1.smithi012.stderr:2023-12-03T11:45:29.349+0000 7f8d65e83640 -1 osd.1 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.355 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:45:29.349+0000 7fcca7b26640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 83869) UID: 0 2023-12-03T11:45:29.355 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:45:29.349+0000 7fcca7b26640 -1 osd.2 253 *** Got signal Terminated *** 2023-12-03T11:45:29.355 INFO:tasks.ceph.osd.2.smithi012.stderr:2023-12-03T11:45:29.349+0000 7fcca7b26640 -1 osd.2 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.355 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:45:29.350+0000 7fa4fe8e2640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 83858) UID: 0 2023-12-03T11:45:29.355 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:45:29.350+0000 7fa4fe8e2640 -1 osd.6 253 *** Got signal Terminated *** 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.6.smithi191.stderr:2023-12-03T11:45:29.350+0000 7fa4fe8e2640 -1 osd.6 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:45:29.350+0000 7f930bc7e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 83874) UID: 0 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:45:29.350+0000 7f930bc7e640 -1 osd.3 253 *** Got signal Terminated *** 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.3.smithi012.stderr:2023-12-03T11:45:29.350+0000 7f930bc7e640 -1 osd.3 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:45:29.351+0000 7fc8fb2a3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 83860) UID: 0 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:45:29.351+0000 7fc8fb2a3640 -1 osd.7 253 *** Got signal Terminated *** 2023-12-03T11:45:29.356 INFO:tasks.ceph.osd.7.smithi191.stderr:2023-12-03T11:45:29.351+0000 7fc8fb2a3640 -1 osd.7 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f5bdd285640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 83545) UID: 0 2023-12-03T11:45:29.357 INFO:tasks.ceph.mon.a.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f5bdd285640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-03T11:45:29.357 INFO:tasks.ceph.mon.c.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f03e2c22640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i c (PID: 83546) UID: 0 2023-12-03T11:45:29.357 INFO:tasks.ceph.mon.c.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f03e2c22640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-03T11:45:29.358 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:45:29.351+0000 7f47d24be640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 83854) UID: 0 2023-12-03T11:45:29.358 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:45:29.351+0000 7f47d24be640 -1 osd.5 253 *** Got signal Terminated *** 2023-12-03T11:45:29.358 INFO:tasks.ceph.osd.5.smithi191.stderr:2023-12-03T11:45:29.351+0000 7f47d24be640 -1 osd.5 253 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-03T11:45:29.358 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f266f70d640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i y (PID: 83585) UID: 0 2023-12-03T11:45:29.358 INFO:tasks.ceph.mgr.y.smithi012.stderr:2023-12-03T11:45:29.351+0000 7f266f70d640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-03T11:45:29.553 INFO:tasks.ceph.mgr.x.smithi191.stderr:daemon-helper: command crashed with signal 15 2023-12-03T11:47:05.450 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:47:05.450 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateAfterToday4AWS4 FAILED 2023-12-03T11:47:05.450 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:212 2023-12-03T11:47:05.451 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:212 2023-12-03T11:47:05.451 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:212 2023-12-03T11:47:55.450 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:47:55.451 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 STARTED 2023-12-03T11:47:55.451 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:47:55.451 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeEpochAWS4 FAILED 2023-12-03T11:47:55.451 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:168 2023-12-03T11:47:55.451 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:168 2023-12-03T11:47:55.452 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:168 2023-12-03T11:48:45.449 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:48:45.450 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 STARTED 2023-12-03T11:48:45.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:48:45.549 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateBeforeToday4AWS4 FAILED 2023-12-03T11:48:45.549 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:258 2023-12-03T11:48:45.549 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:258 2023-12-03T11:48:45.549 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:258 2023-12-03T11:49:35.550 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:49:35.550 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 STARTED 2023-12-03T11:49:35.550 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:49:35.550 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateEmptyAWS4 FAILED 2023-12-03T11:49:35.550 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:398 2023-12-03T11:49:35.551 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:398 2023-12-03T11:49:35.551 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:398 2023-12-03T11:50:25.550 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 STARTED 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateInvalidAWS4 FAILED 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:442 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:442 2023-12-03T11:50:25.551 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:442 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 STARTED 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadDateNoneAWS4 FAILED 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:304 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:304 2023-12-03T11:51:15.550 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:304 2023-12-03T11:52:05.549 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:52:05.549 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 STARTED 2023-12-03T11:52:05.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:52:05.649 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testObjectCreateBadMd5InvalidGarbageAWS4 FAILED 2023-12-03T11:52:05.649 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:586 2023-12-03T11:52:05.649 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:586 2023-12-03T11:52:05.649 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:586 2023-12-03T11:52:55.650 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:52:55.650 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 STARTED 2023-12-03T11:52:55.650 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:52:55.651 INFO:teuthology.orchestra.run.smithi012.stdout:suite > AWS4 tests > AWS4Test.testUploadFileHLAPIBigFileAWS4 FAILED 2023-12-03T11:52:55.651 INFO:teuthology.orchestra.run.smithi012.stdout: com.amazonaws.SdkClientException at AWS4Test.java:825 2023-12-03T11:52:55.651 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: org.apache.http.conn.HttpHostConnectException at AWS4Test.java:825 2023-12-03T11:52:55.651 INFO:teuthology.orchestra.run.smithi012.stdout: Caused by: java.net.ConnectException at AWS4Test.java:825 2023-12-03T11:53:45.649 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:53:45.650 INFO:teuthology.orchestra.run.smithi012.stdout:---------------------------------------------------------------------- 2023-12-03T11:53:45.650 INFO:teuthology.orchestra.run.smithi012.stdout:| Results: FAILURE (26 tests, 18 successes, 8 failures, 0 skipped) | 2023-12-03T11:53:45.650 INFO:teuthology.orchestra.run.smithi012.stdout:---------------------------------------------------------------------- 2023-12-03T11:53:45.749 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.749 INFO:teuthology.orchestra.run.smithi012.stderr:26 tests completed, 8 failed 2023-12-03T11:53:45.749 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:53:45.749 INFO:teuthology.orchestra.run.smithi012.stdout:> Task :test FAILED 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:FAILURE: Build failed with an exception. 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:* What went wrong: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:Execution failed for task ':test'. 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:> There were failing tests. See the report at: file:///home/ubuntu/cephtest/s3-tests-java/build/reports/tests/test/index.html 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:* Try: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.752 INFO:teuthology.orchestra.run.smithi012.stderr:* Get more help at https://help.gradle.org 2023-12-03T11:53:45.753 INFO:teuthology.orchestra.run.smithi012.stdout: 2023-12-03T11:53:45.753 INFO:teuthology.orchestra.run.smithi012.stdout:Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. 2023-12-03T11:53:45.753 INFO:teuthology.orchestra.run.smithi012.stdout:Use '--warning-mode all' to show the individual deprecation warnings. 2023-12-03T11:53:45.753 INFO:teuthology.orchestra.run.smithi012.stdout:See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings 2023-12-03T11:53:45.754 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T11:53:45.754 INFO:teuthology.orchestra.run.smithi012.stderr:BUILD FAILED in 14m 16s 2023-12-03T11:53:45.755 INFO:teuthology.orchestra.run.smithi012.stdout:5 actionable tasks: 4 executed, 1 up-to-date 2023-12-03T11:53:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:53:46.075 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 94, in begin self.run_tests() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 340, in run_tests self.ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' 2023-12-03T11:53:46.150 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=98dd7c9e33bf4d368fc4ac30dfb43e9c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 94, in begin self.run_tests() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 340, in run_tests self.ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test' 2023-12-03T11:53:46.152 DEBUG:teuthology.run_tasks:Unwinding manager s3tests-java 2023-12-03T11:53:46.166 DEBUG:tasks.s3tests_java:S3 Tests Java: END 2023-12-03T11:53:46.166 INFO:tasks.s3tests_java:S3 Tests Java: Cleaning up s3-tests-java... 2023-12-03T11:53:46.166 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/s3-tests-java 2023-12-03T11:53:46.236 INFO:tasks.s3tests_java:S3 Tests Java: Deleting S3 users... 2023-12-03T11:53:46.237 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph 2023-12-03T11:53:46.293 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:53:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T11:58:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-03T11:58:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T11:58:46.297 ERROR:teuthology.run_tasks:Manager failed: s3tests-java Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end self.delete_users(client) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users self.ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T11:58:46.303 DEBUG:teuthology.run_tasks:Unwinding manager ragweed 2023-12-03T11:58:46.313 DEBUG:teuthology.run_tasks:Unwinding manager tox 2023-12-03T11:58:46.323 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf /home/ubuntu/cephtest/tox-venv 2023-12-03T11:58:46.400 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2023-12-03T11:58:46.409 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end self.delete_users(client) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users self.ctx.cluster.only(client).run( File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T11:58:46.412 DEBUG:tasks.rgw.client.0:waiting for process to exit 2023-12-03T11:58:46.412 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T11:58:46.412 INFO:tasks.rgw.client.0:Stopped 2023-12-03T11:58:46.412 DEBUG:teuthology.orchestra.run.smithi012:> rm -f /home/ubuntu/cephtest/rgw.opslog.ceph.client.0.sock 2023-12-03T11:58:46.443 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f /etc/ceph/vault-root-token 2023-12-03T11:58:46.530 INFO:tasks.util.rgw:rgwadmin: client.0 : ['gc', 'process', '--include-all'] 2023-12-03T11:58:46.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'ceph', 'gc', 'process', '--include-all'] 2023-12-03T11:58:46.531 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all 2023-12-03T11:58:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2023-12-03T11:58:46.587 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2023-12-03T12:03:46.589 INFO:teuthology.orchestra.run.smithi012.stderr:2023-12-03T12:03:46.587+0000 7fea3152c800 0 monclient(hunting): authenticate timed out after 300 2023-12-03T12:03:46.590 INFO:teuthology.orchestra.run.smithi012.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-12-03T12:03:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:03:46.592 ERROR:teuthology.run_tasks:Manager failed: rgw Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 226, in start_rgw File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 473, in task File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 300, in create_pools File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/rgw.py", line 242, in start_rgw File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/util/rgw.py", line 34, in rgwadmin File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster ceph gc process --include-all' 2023-12-03T12:03:46.607 DEBUG:teuthology.run_tasks:Unwinding manager openssl_keys 2023-12-03T12:03:46.620 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-03T12:03:46.669 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-03T12:03:46.670 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-03T12:05:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-03T12:05:46.744 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-03T12:05:46.756 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-03T12:05:46.756 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-03T12:05:46.756 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-03T12:05:46.757 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.760 INFO:tasks.ceph.osd.0:Stopped 2023-12-03T12:05:46.760 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-03T12:05:46.760 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.761 INFO:tasks.ceph.osd.1:Stopped 2023-12-03T12:05:46.761 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-03T12:05:46.761 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.761 INFO:tasks.ceph.osd.2:Stopped 2023-12-03T12:05:46.761 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-03T12:05:46.761 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.762 INFO:tasks.ceph.osd.3:Stopped 2023-12-03T12:05:46.762 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-03T12:05:46.762 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.762 INFO:tasks.ceph.osd.4:Stopped 2023-12-03T12:05:46.762 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-03T12:05:46.762 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.763 INFO:tasks.ceph.osd.5:Stopped 2023-12-03T12:05:46.763 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-03T12:05:46.763 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.763 INFO:tasks.ceph.osd.6:Stopped 2023-12-03T12:05:46.763 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-03T12:05:46.763 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.763 INFO:tasks.ceph.osd.7:Stopped 2023-12-03T12:05:46.764 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-03T12:05:46.764 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-03T12:05:46.764 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.764 INFO:tasks.ceph.mgr.y:Stopped 2023-12-03T12:05:46.764 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-03T12:05:46.764 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:05:46.765 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1447, in run_daemon File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi191 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x' 2023-12-03T12:05:46.776 INFO:tasks.ceph.mgr.x:Stopped 2023-12-03T12:05:46.776 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-03T12:05:46.777 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-03T12:05:46.777 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.777 INFO:tasks.ceph.mon.a:Stopped 2023-12-03T12:05:46.777 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-03T12:05:46.777 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.777 INFO:tasks.ceph.mon.c:Stopped 2023-12-03T12:05:46.777 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-03T12:05:46.778 INFO:teuthology.orchestra.run:waiting for 300 2023-12-03T12:05:46.778 INFO:tasks.ceph.mon.b:Stopped 2023-12-03T12:05:46.778 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-03T12:05:46.778 DEBUG:teuthology.orchestra.run.smithi012:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'not have an application enabled' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-03T12:05:46.833 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T12:05:46.833 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-03T12:05:46.939 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T12:05:46.940 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-03T12:05:47.014 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T12:05:47.014 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-03T12:05:47.098 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T12:05:47.098 DEBUG:teuthology.orchestra.run.smithi012:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-03T12:05:47.195 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T12:05:47.195 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-03T12:05:47.325 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T12:05:47.325 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-03T12:05:47.381 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T12:05:47.381 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-03T12:05:47.445 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T12:05:47.445 DEBUG:teuthology.orchestra.run.smithi191:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-03T12:05:47.519 INFO:tasks.ceph:Archiving mon data... 2023-12-03T12:05:47.521 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/data/mon.a.tgz 2023-12-03T12:05:47.521 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-12-03T12:05:47.550 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.NzTb6IEZVW 2023-12-03T12:05:47.551 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.NzTb6IEZVW 2023-12-03T12:05:47.734 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.NzTb6IEZVW 2023-12-03T12:05:47.848 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.NzTb6IEZVW is 356KB 2023-12-03T12:05:47.943 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.NzTb6IEZVW 2023-12-03T12:05:47.970 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/data/mon.c.tgz 2023-12-03T12:05:47.970 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2023-12-03T12:05:48.037 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.OHy2RNSgk5 2023-12-03T12:05:48.037 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OHy2RNSgk5 2023-12-03T12:05:48.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.OHy2RNSgk5 2023-12-03T12:05:48.322 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.OHy2RNSgk5 is 361KB 2023-12-03T12:05:48.417 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.OHy2RNSgk5 2023-12-03T12:05:48.445 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/data/mon.b.tgz 2023-12-03T12:05:48.445 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2023-12-03T12:05:48.463 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.jH5Y81zwfr 2023-12-03T12:05:48.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jH5Y81zwfr 2023-12-03T12:05:48.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.jH5Y81zwfr 2023-12-03T12:05:48.719 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.jH5Y81zwfr is 361KB 2023-12-03T12:05:48.814 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.jH5Y81zwfr 2023-12-03T12:05:48.828 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-03T12:05:48.828 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-03T12:05:48.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-03T12:05:48.923 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-03T12:05:48.950 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-03T12:05:48.950 INFO:tasks.ceph:Archiving crash dumps... 2023-12-03T12:05:48.951 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/lib/ceph/crash to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi012/crash 2023-12-03T12:05:48.952 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-03T12:05:49.046 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/crash to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi191/crash 2023-12-03T12:05:49.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-03T12:05:49.072 INFO:tasks.ceph:Compressing logs... 2023-12-03T12:05:49.073 DEBUG:teuthology.orchestra.run.smithi012:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-03T12:05:49.075 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-03T12:05:49.157 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83483.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83523.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.83483.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83483.log.gz 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83523.log.gzgzip 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83563.log 2023-12-03T12:05:49.158 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83563.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.83563.log.gz -- 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr: /var/log/ceph/ceph-osd.5.log 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83603.log 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr: 90.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83763.log 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83603.log.gz 2023-12-03T12:05:49.159 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83763.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83763.log.gz 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83683.log 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83723.log 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83683.log: /var/log/ceph/ceph-mon.b.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83683.log.gz 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83643.log 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83723.log.gz 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/ceph-client.admin.83643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83643.log.gz 2023-12-03T12:05:49.160 INFO:teuthology.orchestra.run.smithi191.stderr: 88.3% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-03T12:05:49.161 INFO:teuthology.orchestra.run.smithi191.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90240.log 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109943.log 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.109358.log 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90240.log.gz 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.109943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89554.log 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr: 8.2% -- replaced with /var/log/ceph/ceph-client.0.109943.log.gz 2023-12-03T12:05:49.208 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90237.log 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.crash.109358.log: 51.4% -- replaced with /var/log/ceph/ceph-client.crash.109358.log.gz 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100865.log 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89554.log.gz 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90237.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90237.log.gz 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89903.log 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.100865.log.gz 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95990.log 2023-12-03T12:05:49.209 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102177.log 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89903.log.gz 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.95990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105797.log 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95990.log.gz 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99377.log 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr: 81.6% -- replaced with /var/log/ceph/ceph-client.admin.102177.log.gz 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.108679.log 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105797.log.gz 2023-12-03T12:05:49.210 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.99377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110013.log 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.108679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94655.log 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr: 83.9%/var/log/ceph/ceph-client.admin.110013.log: -- replaced with /var/log/ceph/ceph-client.0.108679.log.gz 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.110013.log.gz -5 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101767.log 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83587.log 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.94655.log.gz 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103161.log 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr: 81.5% -- replaced with /var/log/ceph/ceph-client.admin.101767.log.gz 2023-12-03T12:05:49.211 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.83587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83587.log.gz 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90572.log 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105691.log 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.103161.log.gz 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90572.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90572.log.gz 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94843.log 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90580.log 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105691.log: 92.2% -- replaced with /var/log/ceph/ceph-client.0.99377.log.gz 2023-12-03T12:05:49.212 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89680.log 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94843.log.gz 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.105691.log.gz 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91379.log 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90580.log.gz 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93705.log 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89680.log.gz 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96234.log 2023-12-03T12:05:49.213 INFO:teuthology.orchestra.run.smithi012.stderr: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91379.log.gz 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100537.log 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104555.log 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.93705.log.gz 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.96234.log.gz 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91189.log 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.100537.log.gz 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103079.log 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.104555.log.gz 2023-12-03T12:05:49.214 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90495.log 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr: 43.5% -- replaced with /var/log/ceph/ceph-client.0.91189.log.gz 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91083.log 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103079.log.gz 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103571.log 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90495.log.gz 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96120.log 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98894.log 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103571.log.gz 2023-12-03T12:05:49.215 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93623.log 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr: 95.8% -- replaced with /var/log/ceph/ceph-client.0.91083.log.gz 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94737.log 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr: 24.8% -- replaced with /var/log/ceph/ceph-client.admin.98894.log.gz 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92871.log 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr: 81.2% -- replaced with /var/log/ceph/ceph-client.admin.93623.log.gz 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91569.log 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.94737.log.gz 2023-12-03T12:05:49.216 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.92871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97205.log 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.92871.log.gz 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104391.log 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91569.log.gz 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105233.log 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94491.log 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.104391.log.gz 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102341.log 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.105233.log.gz 2023-12-03T12:05:49.217 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105069.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr: 80.1% -- replaced with /var/log/ceph/ceph-client.admin.94491.log.gz 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89498.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102341.log: /var/log/ceph/ceph-client.admin.105069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99963.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.105069.log.gz 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93951.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89498.log.gz 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89897.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.99963.log: /var/log/ceph/ceph-client.admin.93951.log: 81.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90119.log 2023-12-03T12:05:49.218 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99963.log.gz 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.93951.log.gz 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89897.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.102341.log.gz 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97315.log 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89897.log.gz 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90978.log 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90119.log.gz 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105503.log 2023-12-03T12:05:49.219 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91018.log 0.0% 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90978.log.gz 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102915.log 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105503.log.gz 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.91018.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91018.log.gz 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91060.log 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93599.log 2023-12-03T12:05:49.220 INFO:teuthology.orchestra.run.smithi012.stderr: 81.6% -- replaced with /var/log/ceph/ceph-client.admin.102915.log.gz 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.91060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100045.log 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.91060.log.gz 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose 93.2% -- /var/log/ceph/ceph-client.admin.105821.log -- replaced with /var/log/ceph/ceph-client.admin.97205.log.gz 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93599.log.gz 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104145.log 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr: 80.7% -- replaced with /var/log/ceph/ceph-client.admin.100045.log.gz 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109370.log 2023-12-03T12:05:49.221 INFO:teuthology.orchestra.run.smithi012.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.105821.log.gz 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101357.log 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.104145.log.gz 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104473.log 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.109370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109370.log.gz 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105315.log 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.101357.log.gz 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103407.log 2023-12-03T12:05:49.222 INFO:teuthology.orchestra.run.smithi012.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.104473.log.gz 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100455.log 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105315.log: /var/log/ceph/ceph-client.admin.103407.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.105315.log.gz 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105397.log 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103407.log.gz 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100455.log: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.97315.log.gz 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr: 79.9% -- replaced with /var/log/ceph/ceph-client.admin.100455.log.gz 2023-12-03T12:05:49.223 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83997.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.audit.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.105397.log.gz 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96805.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105903.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.83997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83997.log.gz 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93117.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr: 20.3% -- replaced with /var/log/ceph/ceph-client.admin.96805.log.gz 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101275.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98810.log 2023-12-03T12:05:49.224 INFO:teuthology.orchestra.run.smithi012.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.105903.log.gz 81.1% -- replaced with /var/log/ceph/ceph-client.admin.93117.log.gz 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94033.log 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.101275.log.gz 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89638.log 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98810.log: /var/log/ceph/ceph-client.admin.94033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89904.log 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.98810.log.gz 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr: 94.6% 81.5% -- replaced with /var/log/ceph/ceph.audit.log.gz -- replaced with /var/log/ceph/ceph-client.admin.94033.log.gz 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.225 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90573.log 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89638.log.gz 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105773.log 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89904.log.gz 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101603.log 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90573.log.gz 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105773.log.gzgzip 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.102669.log 2023-12-03T12:05:49.226 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101685.log 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.96120.log.gz 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr: 80.8% -- replaced with /var/log/ceph/ceph-client.admin.101603.log.gz 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102669.log: 81.4% -- replaced with /var/log/ceph/ceph-client.admin.102669.log.gz 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93869.log 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93247.log 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93869.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.101685.log.gz 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102751.log 2023-12-03T12:05:49.227 INFO:teuthology.orchestra.run.smithi012.stderr: 80.2% -- replaced with /var/log/ceph/ceph-client.admin.93869.log.gz 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94115.log 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr: 80.1% -- replaced with /var/log/ceph/ceph-client.admin.93247.log.gz 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95656.log 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr: 82.0% -- replaced with /var/log/ceph/ceph-client.admin.102751.log.gz 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99799.log 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr: 82.3% -- replaced with /var/log/ceph/ceph-client.admin.94115.log.gz 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.95656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93493.log 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr: 57.1% -- replaced with /var/log/ceph/ceph-client.admin.95656.log.gz 2023-12-03T12:05:49.228 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.99799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94361.log 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr: 82.4% -- replaced with /var/log/ceph/ceph-client.admin.99799.log.gz 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93223.log 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.93493.log.gz 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94279.log 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.94361.log.gz 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93035.log 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93223.log.gz 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94279.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-03T12:05:49.229 INFO:teuthology.orchestra.run.smithi012.stderr: 79.9% -- replaced with /var/log/ceph/ceph-client.admin.94279.log.gz 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97865.log 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.93035.log.gz 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101193.log 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83878.log 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103243.log 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr: 85.2% -- replaced with /var/log/ceph/ceph-client.admin.101193.log.gz 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.83878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90772.log 2023-12-03T12:05:49.230 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.83878.log.gz 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102259.log 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103243.log: /var/log/ceph/ceph-client.admin.90772.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90772.log.gz 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96178.log 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.103243.log.gz 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.97865.log.gz 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100947.log 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96178.log: gzip 81.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.102259.log.gz --verbose 2023-12-03T12:05:49.231 INFO:teuthology.orchestra.run.smithi012.stderr: -- /var/log/ceph/ceph-client.admin.98643.log 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr: 55.3% -- replaced with /var/log/ceph/ceph-client.admin.96178.log.gz 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103325.log 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.100947.log.gz 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103653.log 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100127.log 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103325.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.98643.log.gz 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103653.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.103325.log.gz 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105048.log 2023-12-03T12:05:49.232 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99039.log 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100127.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.103653.log.gz 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90455.log 0.0% 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.105048.log.gz 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.100127.log.gz 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102423.log 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.99039.log: /var/log/ceph/ceph-client.admin.90455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94867.log 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90455.log.gz 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr: 84.0% -- replaced with /var/log/ceph/ceph-client.0.99039.log.gz 2023-12-03T12:05:49.233 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.crash.smithi012.109346.log 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.102423.log.gz 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100209.log 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.admin.94867.log.gz 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.crash.smithi012.109346.log: 13.0% -- replaced with /var/log/ceph/ceph-client.crash.smithi012.109346.log.gzgzip 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.105479.log 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90238.log 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100209.log: /var/log/ceph/ceph-client.admin.105479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102013.log 2023-12-03T12:05:49.234 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105479.log.gz 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.100209.log.gz 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90238.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90238.log.gz 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102013.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.77724.log 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.102013.log.gz 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89895.log 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph.tmp-client.admin.77724.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.77724.log.gz 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94197.log 2023-12-03T12:05:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94467.log 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89895.log.gz 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97975.log 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr: 80.1% -- replaced with /var/log/ceph/ceph-client.admin.94197.log.gz 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103817.log 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94467.log.gz 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104883.log 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101111.log 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103817.log.gz 2023-12-03T12:05:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104965.log 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.104883.log.gz 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96903.log 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr: 82.1% -- replaced with /var/log/ceph/ceph-client.admin.101111.log.gz 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.97975.log.gz 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98474.log 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr: 80.3% -- replaced with /var/log/ceph/ceph-client.admin.104965.log.gz 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101849.log 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104637.log 2023-12-03T12:05:49.237 INFO:teuthology.orchestra.run.smithi012.stderr: 84.1% -- replaced with /var/log/ceph/ceph-client.admin.96903.log.gz 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98474.log: /var/log/ceph/ceph-client.admin.101849.log: 80.5% -- replaced with /var/log/ceph/ceph-client.admin.98474.log.gz 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102095.log 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.101849.log.gz 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90456.log 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.104637.log.gz 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.103489.log 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102095.log: /var/log/ceph/ceph-client.admin.90456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93329.log 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.102095.log.gz 2023-12-03T12:05:49.238 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90456.log.gz 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99475.log 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94949.log 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.93329.log.gz 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.99475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90936.log 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr: 80.0% -- replaced with /var/log/ceph/ceph-client.0.99475.log.gz 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90231.log 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.103489.log.gz 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90936.log: 85.7% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94949.log.gz 2023-12-03T12:05:49.239 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90936.log.gz 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89722.log 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.admin.log 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90231.log.gz 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103735.log 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89722.log.gz 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ops-log-ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100701.log 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103735.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.103735.log.gz 2023-12-03T12:05:49.240 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99881.log 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.100701.log.gz 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93575.log 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.99881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89900.log 81.8% -- replaced with /var/log/ceph/ceph-client.admin.99881.log.gz 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93575.log.gz 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103981.log 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89900.log.gz 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90239.log 2023-12-03T12:05:49.241 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104063.log 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.103981.log.gz 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90239.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90239.log.gz -5 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.108821.log 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99325.log 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104063.log: 94.3% -- replaced with /var/log/ceph/ops-log-ceph-client.admin.log.gz 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93199.log 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.104063.log.gz 2023-12-03T12:05:49.242 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.108821.log: /var/log/ceph/ceph-client.admin.99325.log: 50.5% -- replaced with /var/log/ceph/ceph-client.admin.99325.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90236.log 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89899.log 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.93199.log.gz 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95766.log 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90236.log.gz 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105609.log 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89899.log.gz 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.95766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102833.log 2023-12-03T12:05:49.243 INFO:teuthology.orchestra.run.smithi012.stderr: 54.6% -- replaced with /var/log/ceph/ceph-client.admin.95766.log.gz 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98085.log 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.105609.log.gz 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102833.log: 83.0% -- replaced with /var/log/ceph/ceph-client.admin.102833.log.gz 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.108821.log.gz 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97645.log 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89902.log 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90120.log 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89902.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104227.log 2023-12-03T12:05:49.244 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.98085.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90120.log.gz/var/log/ceph/ceph-client.admin.104227.log: 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96064.log 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr: 81.6% -- replaced with /var/log/ceph/ceph-client.admin.104227.log.gz 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr: 88.5%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.97645.log.gz 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102997.log 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96064.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.96064.log.gz 2023-12-03T12:05:49.245 INFO:teuthology.orchestra.run.smithi012.stderr: --verbose -- /var/log/ceph/ceph-client.admin.90856.log 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102997.log: gzip -5 --verbose -- /var/log/ceph/ops-log-ceph-client.0.log 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.102997.log.gz 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90856.log.gz 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104801.log 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose/var/log/ceph/ops-log-ceph-client.0.log: -- /var/log/ceph/ceph-client.admin.93787.log 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90575.log 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr: 81.4% -- replaced with /var/log/ceph/ceph-client.admin.104801.log.gz 2023-12-03T12:05:49.246 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91661.log 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90575.log: 81.5% -- replaced with /var/log/ceph/ceph-client.admin.93787.log.gz 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90575.log.gz 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92953.log 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102587.log 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91661.log.gz 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.92953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94819.log 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102587.log: 80.3% -- replaced with /var/log/ceph/ceph-client.admin.92953.log.gz 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.102587.log.gz 2023-12-03T12:05:49.247 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98556.log 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93411.log 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.98556.log: /var/log/ceph/ceph-client.admin.94819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94819.log.gz 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr: 83.8% -- replaced with /var/log/ceph/ceph-client.admin.98556.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.0.99137.log 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89596.log 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.93411.log: /var/log/ceph/ceph-client.0.99137.log: 79.8% 84.1% -- replaced with /var/log/ceph/ceph-client.0.99137.log.gz 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93411.log.gz 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104719.log 2023-12-03T12:05:49.248 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89596.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89596.log.gz 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91281.log 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104719.log: /var/log/ceph/ceph.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.104719.log.gz 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101029.log 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91281.log: 80.0% -- replaced with /var/log/ceph/ceph-client.0.91281.log.gz 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101029.log: 81.6% -- replaced with /var/log/ceph/ceph-client.admin.101029.log.gz 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose 94.6% -- /var/log/ceph/ceph-client.admin.97755.log 2023-12-03T12:05:49.249 INFO:teuthology.orchestra.run.smithi012.stderr: -- replaced with /var/log/ceph/ops-log-ceph-client.0.log.gz 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105985.log 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97755.log: 93.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91471.log 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105985.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.105985.log.gz 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103899.log 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91471.log: 80.1% -- replaced with /var/log/ceph/ceph-client.0.91471.log.gz 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95390.log 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.103899.log: 84.8% -- replaced with /var/log/ceph/ceph-client.admin.103899.log.gz 2023-12-03T12:05:49.250 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104309.log 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.95390.log: 60.3% -- replaced with /var/log/ceph/ceph-client.admin.95390.log.gz 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108922.log 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.104309.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.104309.log.gz 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94443.log 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.108922.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.108922.log.gz 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/rgw.ceph.client.0.log 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.94443.log.gz 2023-12-03T12:05:49.251 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105527.log 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/rgw.ceph.client.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100373.log 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105527.log: 81.8% -- replaced with /var/log/ceph/ceph-client.admin.105527.log.gz 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.97755.log.gz 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97425.log 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100373.log: 81.3% -- replaced with /var/log/ceph/ceph-client.admin.100373.log.gz 2023-12-03T12:05:49.252 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90814.log 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97425.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.97425.log.gz 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91753.log 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90814.log.gz 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89893.log 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91753.log: 42.8% -- replaced with /var/log/ceph/ceph-client.0.91753.log.gz 2023-12-03T12:05:49.270 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89398.log 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89893.log.gz 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94573.log 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.89398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.89398.log.gz 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101931.log 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.94573.log: 80.0% -- replaced with /var/log/ceph/ceph-client.admin.94573.log.gz 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101439.log 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101931.log: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.101931.log.gz 2023-12-03T12:05:49.271 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100291.log 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101439.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.101439.log.gz 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100783.log 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100291.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.100291.log.gz 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100619.log 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100783.log: 81.1% -- replaced with /var/log/ceph/ceph-client.admin.100783.log.gz 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90896.log 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.100619.log: 81.9% -- replaced with /var/log/ceph/ceph-client.admin.100619.log.gz 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.91845.log 2023-12-03T12:05:49.272 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90896.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90896.log.gz 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90496.log 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.91845.log: 43.2% -- replaced with /var/log/ceph/ceph-client.0.91845.log.gz 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101521.log 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.90496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.90496.log.gz 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95031.log 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.101521.log: 81.5% -- replaced with /var/log/ceph/ceph-client.admin.101521.log.gz 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102505.log 2023-12-03T12:05:49.273 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.95031.log: 81.7% -- replaced with /var/log/ceph/ceph-client.admin.95031.log.gz 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105151.log 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.102505.log: 81.2% -- replaced with /var/log/ceph/ceph-client.admin.102505.log.gz 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96985.log 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.105151.log: 84.6% -- replaced with /var/log/ceph/ceph-client.admin.105151.log.gz 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109851.log 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.96985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97095.log 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.0.109851.log: 8.2% -- replaced with /var/log/ceph/ceph-client.0.109851.log.gz 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.96985.log.gz 2023-12-03T12:05:49.274 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97535.log 2023-12-03T12:05:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97095.log: 77.7% -- replaced with /var/log/ceph/ceph-client.admin.97095.log.gz 2023-12-03T12:05:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-03T12:05:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-client.admin.97535.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97535.log.gz 2023-12-03T12:05:49.275 INFO:teuthology.orchestra.run.smithi012.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98726.log 2023-12-03T12:05:49.276 INFO:teuthology.orchestra.run.smithi012.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.98726.log: 80.4% -- replaced with /var/log/ceph/ceph-client.admin.98726.log.gz 2023-12-03T12:05:49.393 INFO:teuthology.orchestra.run.smithi012.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-03T12:05:49.572 INFO:teuthology.orchestra.run.smithi012.stderr: 91.0% -- replaced with /var/log/ceph/rgw.ceph.client.0.log.gz 2023-12-03T12:05:49.657 INFO:teuthology.orchestra.run.smithi191.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-03T12:05:49.917 INFO:teuthology.orchestra.run.smithi012.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-03T12:05:50.259 INFO:teuthology.orchestra.run.smithi012.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-03T12:05:51.916 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-03T12:05:52.283 INFO:teuthology.orchestra.run.smithi012.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-03T12:05:52.353 INFO:teuthology.orchestra.run.smithi191.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-03T12:05:52.753 INFO:teuthology.orchestra.run.smithi012.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-03T12:05:53.552 INFO:teuthology.orchestra.run.smithi191.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-03T12:05:54.740 INFO:teuthology.orchestra.run.smithi012.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-03T12:05:55.754 INFO:teuthology.orchestra.run.smithi191.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-03T12:05:55.756 INFO:teuthology.orchestra.run.smithi191.stderr: 2023-12-03T12:05:55.756 INFO:teuthology.orchestra.run.smithi191.stderr:real 0m6.628s 2023-12-03T12:05:55.756 INFO:teuthology.orchestra.run.smithi191.stderr:user 0m16.941s 2023-12-03T12:05:55.756 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m0.594s 2023-12-03T12:06:02.117 INFO:teuthology.orchestra.run.smithi012.stderr: 95.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-03T12:06:02.119 INFO:teuthology.orchestra.run.smithi012.stderr: 2023-12-03T12:06:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:real 0m13.018s 2023-12-03T12:06:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:user 0m26.917s 2023-12-03T12:06:02.119 INFO:teuthology.orchestra.run.smithi012.stderr:sys 0m1.016s 2023-12-03T12:06:02.119 INFO:tasks.ceph:Archiving logs... 2023-12-03T12:06:02.120 DEBUG:teuthology.misc:Transferring archived files from smithi012:/var/log/ceph to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi012/log 2023-12-03T12:06:02.120 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-03T12:06:05.679 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi191/log 2023-12-03T12:06:05.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-03T12:06:08.135 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1918, in task File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph.py", line 1928, in task File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2758, in wait_for_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2590, in get_num_active_clean File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2023-12-03T12:06:08.148 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-03T12:06:08.158 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 101, in end File "/home/teuthworker/src/git.ceph.com_ceph-c_bcbf278cf395873c2a482f5a7b2345faf2160319/qa/tasks/s3tests_java.py", line 386, in delete_users File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:06:08.170 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-03T12:06:08.170 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-03T12:06:08.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-03T12:06:08.268 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-03T12:06:08.279 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-03T12:06:08.279 DEBUG:teuthology.orchestra.run.smithi012:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T12:06:08.281 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-03T12:06:08.298 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2023-12-03T12:06:08.302 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T12:06:08.302 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-12-03T12:06:08.302 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 74 +2899us[+2951us] +/- 100ms 2023-12-03T12:06:08.302 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -820us[ -769us] +/- 77ms 2023-12-03T12:06:08.303 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 77 -1461us[-1410us] +/- 84ms 2023-12-03T12:06:08.303 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T12:06:08.309 INFO:teuthology.orchestra.run.smithi012.stderr:bash: line 1: ntpq: command not found 2023-12-03T12:06:08.313 INFO:teuthology.orchestra.run.smithi012.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-03T12:06:08.314 INFO:teuthology.orchestra.run.smithi012.stdout:=============================================================================== 2023-12-03T12:06:08.314 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 11 +2865us[+2862us] +/- 101ms 2023-12-03T12:06:08.314 INFO:teuthology.orchestra.run.smithi012.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -828us[ -831us] +/- 77ms 2023-12-03T12:06:08.314 INFO:teuthology.orchestra.run.smithi012.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -1494us[-1497us] +/- 85ms 2023-12-03T12:06:08.314 INFO:teuthology.orchestra.run.smithi012.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-03T12:06:08.315 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-03T12:06:08.325 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-03T12:06:08.326 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-03T12:06:08.336 DEBUG:teuthology.orchestra.run.smithi012:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-03T12:06:08.476 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-03T12:06:08.574 DEBUG:teuthology.orchestra.run.smithi012:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T12:06:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:06:08.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-03T12:06:08.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:06:08.656 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-03T12:06:08.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-03T12:06:08.677 INFO:teuthology.task.internal:Duration was 3035.938677 seconds 2023-12-03T12:06:08.677 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-03T12:06:08.688 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-03T12:06:08.688 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-03T12:06:08.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-03T12:06:08.741 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T12:06:08.753 INFO:teuthology.orchestra.run.smithi012.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-03T12:06:08.977 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-03T12:06:08.977 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi012.front.sepia.ceph.com 2023-12-03T12:06:08.977 DEBUG:teuthology.orchestra.run.smithi012:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-03T12:06:09.010 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2023-12-03T12:06:09.010 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-03T12:06:09.039 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-03T12:06:09.039 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-03T12:06:09.052 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-03T12:06:09.107 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-03T12:06:09.107 DEBUG:teuthology.orchestra.run.smithi012:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-03T12:06:09.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-03T12:06:09.238 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-03T12:06:09.250 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-03T12:06:09.250 DEBUG:teuthology.orchestra.run.smithi012:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-03T12:06:09.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-03T12:06:09.320 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-03T12:06:09.332 DEBUG:teuthology.orchestra.run.smithi012:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:06:09.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:06:09.376 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = core 2023-12-03T12:06:09.385 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2023-12-03T12:06:09.429 DEBUG:teuthology.orchestra.run.smithi012:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:06:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:06:09.463 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-03T12:06:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-03T12:06:09.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-03T12:06:09.495 INFO:teuthology.task.internal:Transferring archived files... 2023-12-03T12:06:09.496 DEBUG:teuthology.misc:Transferring archived files from smithi012:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi012 2023-12-03T12:06:09.496 DEBUG:teuthology.orchestra.run.smithi012:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-03T12:06:09.557 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128/remote/smithi191 2023-12-03T12:06:09.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-03T12:06:09.592 INFO:teuthology.task.internal:Removing archive directory... 2023-12-03T12:06:09.592 DEBUG:teuthology.orchestra.run.smithi012:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-03T12:06:09.595 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-03T12:06:09.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-03T12:06:09.655 INFO:teuthology.task.internal:Not uploading archives. 2023-12-03T12:06:09.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-03T12:06:09.666 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-03T12:06:09.666 DEBUG:teuthology.orchestra.run.smithi012:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-03T12:06:09.669 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-03T12:06:09.693 INFO:teuthology.orchestra.run.smithi012.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 12:06 /home/ubuntu/cephtest 2023-12-03T12:06:09.696 INFO:teuthology.orchestra.run.smithi191.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 3 12:06 /home/ubuntu/cephtest 2023-12-03T12:06:09.697 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-03T12:06:09.709 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-03T12:06:09.722 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi012 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid tester.client.0 --purge-data --cluster ceph' 2023-12-03T12:06:09.722 INFO:teuthology.nuke:Checking targets against current locks 2023-12-03T12:06:09.748 DEBUG:teuthology.nuke:shortname: smithi012 2023-12-03T12:06:09.748 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T12:06:09.772 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi012.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:05:56.491571', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOYDMCHNBhjGlWzbayJybiaYC93qeZv13jHyTh7c0lvz5KCXrt5k7tAECc6ZHM9rI8bMxWCMauhZfsEvkVYHPEU='} 2023-12-03T12:06:09.775 DEBUG:teuthology.nuke:shortname: smithi191 2023-12-03T12:06:09.775 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-03T12:06:09.795 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/alimasa-2023-12-03_11:04:43-rgw-wip-alimasa-multi-pubsub-etag-distro-default-smithi/7476128', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-03 11:05:56.490166', 'locked_by': 'scheduled_alimasa@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBZ+1QdwWJLvtMSWw+KNzg1rIK0cQ9aN/t7GZd/tSC1IjlZ68HUn7gFiinnZ4aJuc/FBuzhStjVN2OcEXlQtDeQ='} 2023-12-03T12:06:09.822 INFO:teuthology.orchestra.console.smithi012:Power off 2023-12-03T12:06:09.822 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T12:06:09.846 DEBUG:teuthology.orchestra.console.smithi012:power off output: Chassis Power Control: Down/Off 2023-12-03T12:06:09.846 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:09.862 INFO:teuthology.orchestra.console.smithi191:Power off 2023-12-03T12:06:09.863 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-03T12:06:09.871 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-12-03T12:06:09.883 DEBUG:teuthology.orchestra.console.smithi191:power off output: Chassis Power Control: Down/Off 2023-12-03T12:06:09.884 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:09.905 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-03T12:06:13.872 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:13.906 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:13.983 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is on 2023-12-03T12:06:14.016 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-03T12:06:17.988 DEBUG:teuthology.orchestra.console.smithi012:pexpect command: ipmitool -H smithi012.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:18.018 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:18.098 DEBUG:teuthology.orchestra.console.smithi012:check power output: Chassis Power is off 2023-12-03T12:06:18.126 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is on 2023-12-03T12:06:18.199 INFO:teuthology.orchestra.console.smithi012:Power off completed 2023-12-03T12:06:22.128 DEBUG:teuthology.orchestra.console.smithi191:pexpect command: ipmitool -H smithi191.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-03T12:06:22.238 DEBUG:teuthology.orchestra.console.smithi191:check power output: Chassis Power is off 2023-12-03T12:06:22.339 INFO:teuthology.orchestra.console.smithi191:Power off completed 2023-12-03T12:06:22.443 INFO:teuthology.run:Summary data: description: rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} duration: 3035.9386773109436 failure_reason: 'Command failed on smithi012 with status 1: ''cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests AWS4Test''' flavor: default owner: scheduled_alimasa@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=98dd7c9e33bf4d368fc4ac30dfb43e9c status: fail success: false 2023-12-03T12:06:22.444 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-03T12:06:22.518 INFO:teuthology.run:FAIL